Memory consumption keeps increasing

1. Symptoms

• free -h shows available memory shrinking over hours.
• A node is killed by the out-of-memory killer, or the system starts swapping and slows down.
• Performance degrades steadily during a long run and is restored by a restart.

2. Possible causes

• Memory leak in a node or library (objects allocated per message and never freed).
• Unbounded queue or buffer: a slow subscriber with a deep or unlimited queue, or an accumulating list in your own code.
• Growing map: SLAM or costmap memory grows with the explored area, legitimately.
• Large messages kept alive by references (images or clouds stored in history).
• Logging or recording buffered in memory faster than it is written.
• Page cache (not a leak): the OS caches files in otherwise free memory.

3. Measurements to collect

• Resident memory (RSS) per process over time.
• System available memory (not just “free”), swap use.
• Queue depths and message backlog per topic.
• Map size or number of map elements over time.
• Recording buffer and disk write speed.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

free -h                                              # look at "available", not "free" (page cache counts as available)
ps -eo pid,rss,comm --sort=-rss | head -15           # biggest processes by resident memory (kB)
watch -n 10 'ps -o pid,rss,comm -p <PID>'            # one process over time
vmstat 5                                             # si/so columns: swapping
ros2 topic hz /points                                # is the subscriber keeping up with the publisher?
dmesg | grep -i -E "out of memory|oom"               # was something killed?

5. Step-by-step investigation

1. Distinguish page cache from real use: look at “available” in free -h. A large “buff/cache” is normal.
2. Find the owner: list processes by RSS and watch the top few for 30 minutes. A steady upward line in one process is the suspect.
3. Is it a legitimate map? Compare growth with distance travelled; growth that stops when the robot stops exploring is a map, not a leak.
4. Is it a backlog? If the process subscribes to a fast topic and processes slowly, check the queue depth and callback time; a bounded queue stops growth, an unbounded one does not.
5. Look for accumulation in your own code: lists, caches or history buffers that grow with every message.
6. Try a memory tool (a leak checker or allocator profiler) on the suspect node in a test run, with a short representative input.
7. Check recording: memory that grows only while recording points at the writer falling behind.

6. Corrective actions

• Fix or upgrade the leaking component; as a mitigation restart the node periodically with a supervisor.
• Bound queues and buffers; process or drop old messages.
• Cap or prune maps, or use a map format with bounded memory for long runs.
• Avoid storing large messages; copy only what is needed.
• Give the recorder faster storage, or compress or record fewer topics.

7. Validation after the fix

• RSS of each process is flat (or grows only as the map legitimately grows) over a run at least as long as the longest mission.
• Available memory stays above your margin (for example 30 % of RAM).
• No out-of-memory kills or swap activity in the run.
• The fix is verified on the full pipeline at full load.

References

• See the throughput guide for estimating buffer and map memory, and the profiling guide for the measurement routine.

• Real-Time Performance and Resource Constraints SLAM · Lesson