FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 05

Engineering Guides · Module 05

Performance profiling guide

Measure before optimising. A short checklist of commands tells you whether the limit is a core, the GPU, memory, heat or the data itself.

  • Advanced
  • Module 05: Onboard Computer, CPU/GPU Selection & Compute Sizing
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-05

Assumptions: Linux, ROS 2 Jazzy. The classification rule is a first-pass heuristic with fixed thresholds you should adapt; it is executed by the site’s tests. The shell and ROS 2 commands are not. Platform-specific commands are marked.

1. Engineering problem

“It is slow” has many causes. Optimising the wrong stage wastes weeks. A fixed measurement routine, run at full load, finds the limiting resource first.

2. When is this needed?

  • Rates drop or latency grows.
  • Choosing or changing a computer.
  • Before and after any optimisation.

3. Selection and design decisions

ResourceCommandsWhat a problem looks like
CPU coreshtop, top, mpstat -P ALL 1 (sysstat), top -H -p <PID>One core at 100 % while others idle: a serial stage. All cores high: too much total work
Memoryfree -h, vmstat 1Available memory near zero, swap activity (si/so in vmstat) or growth over time (leak)
GPU (NVIDIA)nvidia-smi, nvidia-smi dmonLow GPU use with a busy CPU core: the GPU is starved; high memory use near the limit
Jetson (platform-specific)tegrastatsCPU, GPU, memory, temperature and power on Jetson only
Thermal and frequencygrep MHz /proc/cpuinfo, cat /sys/class/thermal/thermal_zone*/temp, nvidia-smi -q -d PERFORMANCEClock speeds falling as temperature rises: throttling
Data rateros2 topic hz <t>, ros2 topic bw <t>Rate below the publisher’s rate; bandwidth near the link capacity
Diskiostat -x 1 (sysstat)High utilisation and waits while recording

Only use platform-specific commands where the platform supports them: tegrastats exists on NVIDIA Jetson, nvidia-smi needs the NVIDIA driver.

4. Advantages and disadvantages

  • Sampling tools (top, nvidia-smi) are cheap and coarse; profilers are precise and perturb the system.
  • Running at full load shows the real limit and risks stressing the robot: do it with the robot safe.
  • Per-stage timing needs code instrumentation but gives the budget that matters.

5. Requirements and constraints

  • Measure under the real workload and the real power mode.
  • Record the measurement conditions (power mode, ambient temperature, data set).
  • Keep other applications and viewers off the robot while profiling.

6. Architecture

workload ─▶ [ htop / nvidia-smi / free / thermal ] ─▶ classify ─▶ pick ONE fix ─▶ re-measure the same way

7. Implementation

  1. Run the system at full load and let it warm up for 10 minutes.
  2. Capture CPU per core, GPU, memory, temperature and frequency, and topic rates, at the same time.
  3. Classify with the rule below, then confirm with per-stage timing.
  4. Change one thing; repeat the same measurement.
def classify(cores, gpu, mem):
    """cores: per-core CPU use (%), gpu: GPU use (%) or None, mem: memory use (%)."""
    out = []
    if max(cores) >= 90 and sum(cores) / len(cores) < 50:
        out.append("one saturated core: a serial stage limits the pipeline")
    if gpu is not None and gpu < 50 and max(cores) >= 90:
        out.append("GPU starved: a CPU-side stage is limiting")
    if mem >= 90:
        out.append("memory pressure: expect swapping and stalls")
    return out or ["no obvious resource bottleneck: look at latency and queues"]

print(classify([95, 20, 15, 10], 30, 60))
print(classify([60, 55, 58, 62], 20, 95))
print(classify([40, 35, 30, 25], 85, 50))

Printed (checked by the site’s tests):

['one saturated core: a serial stage limits the pipeline', 'GPU starved: a CPU-side stage is limiting']
['memory pressure: expect swapping and stalls']
['no obvious resource bottleneck: look at latency and queues']

Reading it: case 1 has one core at 95 % and the GPU at 30 %, so a serial CPU stage limits the pipeline and starves the GPU; case 2 has memory at 95 %; case 3 has no saturated resource, so the problem is probably queueing, latency or QoS (Module 17). The thresholds (90 %, 50 %) are starting values, not standards.

Estimate expected numbers first with the CPU/GPU Resource Estimator, then compare.

8. Real examples

  • SLAM node at 100 % of one core: a single-threaded stage; reduce its input or choose a faster core.
  • Rates fall after 20 minutes: thermal throttling; check frequency and temperature.
  • Memory grows steadily: a leak or unbounded buffer (see the memory diagnostic when available).

9. Failure scenarios

  • Profiling at idle.
  • Optimising a stage that is not on the critical path.
  • Averaging utilisation over cores and missing a saturated one.
  • Ignoring throttling when comparing runs.

10. Troubleshooting

  1. Check frequency and temperature first: throttling invalidates every other number.
  2. Look at per-core use, not the average.
  3. Check whether the GPU is waiting for the CPU.
  4. Check memory trend over time.
  5. Check topic rates and bandwidth against expectation.

11. Validation

  • Before and after measurements use the same procedure and conditions.
  • The fix improves end-to-end rate or latency, not only one counter.
  • Utilisation margins hold at full load for an hour.
  • The classifier example prints the values above (executed by the site tests).

12. Engineering checklist

13. Further references

  • Linux tool manuals (man top, man vmstat, man mpstat, man iostat) and the NVIDIA nvidia-smi documentation. Not fetched here.

Related on FixTheRobot