Profiling CPU, GPU and memory

1. Engineering problem

A robot that falls behind usually has one saturated thread or a process that grows until the system fails. Total CPU percentage shows neither: look per thread and per process over time.

2. When is this needed?

• A node cannot keep up.
• The robot fails after hours (see the troubleshooting page on long runs).
• Sizing a computer.

3. Selection and design decisions

Linux tools: top -H (per thread), htop, pidstat, perf, /proc/<pid>/status for memory; on NVIDIA Jetson, tegrastats reports CPU, GPU and memory; nvidia-smi on discrete NVIDIA GPUs. A flat per-process memory trend is healthy; a steady slope is a leak or an unbounded buffer. A thread at near 100 % of one core is a bottleneck even when the machine total looks low.

4. Advantages and disadvantages

• Sampling profilers are cheap and statistical; tracing is exact and heavier.
• Longer observation shows trends and costs time.
• Profiling on the target computer matters: a workstation hides problems.

5. Requirements and constraints

• Observation under realistic load and long enough to see trends.
• CPU and memory logged with timestamps.
• Results from the target hardware.

6. Architecture

processes and threads ─▶ per-thread CPU, per-process memory, GPU use (logged over time) ─▶ bottleneck and leak candidates

7. Implementation

1. Log per-process CPU and memory every minute under full load.
2. Find threads near 100 % of a core.
3. Fit the memory trend per process (below).
4. Look at GPU utilisation next to CPU.
5. Fix the largest item and repeat.

def slope(ts, ys):
    n = len(ts); mt, my = sum(ts) / n, sum(ys) / n
    return sum((t - mt) * (y - my) for t, y in zip(ts, ys)) / sum((t - mt) ** 2 for t in ts)
t = [0, 60, 120, 180, 240, 300, 360]                                       # seconds
rss = {"perception": [510, 512, 511, 513, 512, 511, 512], "mapper": [300, 318, 335, 352, 371, 388, 405], "logger": [80, 81, 80, 82, 81, 80, 81]}   # MB
for name, ys in rss.items():
    s = slope(t, ys) * 3600
    print(f"{name:11s} memory trend {s:7.1f} MB/hour", "-> growing: investigate" if s > 20 else "-> flat")
threads = {"image_proc": 38.0, "slam_front_end": 92.0, "slam_back_end": 14.0, "rviz": 21.0}
total = sum(threads.values())
busiest = max(threads, key=threads.get)
print(f"busiest thread: {busiest} at {threads[busiest]:.0f} % of one core; total {total:.0f} % of one core ({total / 4:.0f} % of a 4-core machine)")
print("one core saturated:" , [k for k, v in threads.items() if v > 85])

Printed (checked by the site’s tests):

perception  memory trend    10.7 MB/hour -> flat
mapper      memory trend  1052.1 MB/hour -> growing: investigate
logger      memory trend     4.3 MB/hour -> flat
busiest thread: slam_front_end at 92 % of one core; total 165 % of one core (41 % of a 4-core machine)
one core saturated: ['slam_front_end']

Reading it: the mapper’s memory grows about 1 GB per hour while the other processes are flat (perception 10.7 MB/h, logger 4.3 MB/h); the slam front-end thread uses 92 % of one core, saturating it, while the total is 41 % of a 4-core machine. The numbers are illustrative.

8. Real examples

• Mapping node growth: a map that never forgets old data; bounded after limiting the retained submaps.

9. Failure scenarios

• Looking only at total CPU.
• Profiling on a workstation.
• A short test that misses a slow leak.

10. Troubleshooting

1. Run top -H for the busiest process.
2. Plot memory against time.
3. Check for unbounded queues and growing containers.

11. Validation

• The analysis prints the lines shown (executed by the site tests).
• No process shows a memory trend above the limit over a run as long as the longest mission.

12. Engineering checklist

• Per-thread CPU
• Memory trend per process
• GPU utilisation
• Target hardware
• Long enough run
• Fixes verified

13. Further references

• Linux and NVIDIA tool documentation (top, pidstat, perf, tegrastats, nvidia-smi).

Related on FixTheRobot