Skip to the content
FixTheRobot Explore Robotics / UAV / Module 7 / M7-L4

UAV Systems Engineering · M7-L4 · Hands-on lab (runs on your machine)

Measure CPU and GPU utilization

Turn “it seems fine” into a table with a pass or fail.

  • Embedded Systems, Timing and System Integration
  • Implementation-specific
  • About 60 minutes

What you will do

You will sample CPU (per process and per core), GPU, memory and temperature while the whole stack runs a mission-representative workload (a rosbag replay at real time, or the vehicle on the bench with its sensors), then compute means, percentiles and the combined peak, and compare them with a headroom requirement from your compute budget (lesson M7-T12). The reference analysis runs on synthetic samples; collect your own with the commands below and adapt the loader.

Before you start

  • Lessons M7-T3 (workload management) and M7-T12 (compute and memory budgets).
  • A companion computer with the stack running, or synthetic data.
  • Python 3 with NumPy (and psutil if you want to sample from Python).

Collect

Sample once per second for the full duration of a representative run (at least ten minutes, ideally a full mission). Using psutil to sample per-process CPU and memory, and the platform tool for the GPU (on Jetson, tegrastats; on discrete NVIDIA GPUs, nvidia-smi):

import psutil, time, csv
def sample(names, duration_s, path):
    procs = {p.pid: p for p in psutil.process_iter(['name', 'cmdline']) if any(n in ' '.join(p.info['cmdline'] or []) or n == p.info['name'] for n in names)}
    for p in procs.values(): p.cpu_percent(None)                                    # prime the counters
    with open(path, 'w', newline='') as f:
        w = csv.writer(f); w.writerow(['t', 'pid', 'name', 'cpu_percent', 'rss_mb', 'system_cpu_percent'])
        t0 = time.monotonic()
        while time.monotonic() - t0 < duration_s:
            time.sleep(1.0); sys_cpu = psutil.cpu_percent(None)
            for pid, p in procs.items():
                try: w.writerow([round(time.monotonic() - t0, 2), pid, p.name(), p.cpu_percent(None), round(p.memory_info().rss / 2**20, 1), sys_cpu])
                except psutil.NoSuchProcess: pass
Per-process sampling with psutil. cpu_percent is relative to one core, so values above 100 are possible. Syntax-checked only.
nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu --format=csv -l 1 > gpu.csv     # discrete GPU
# on Jetson: sudo tegrastats --interval 1000 > tegrastats.log   (format differs by JetPack version)
GPU sampling at 1 Hz.

Analysis script (synthetic data)

import numpy as np
rng = np.random.default_rng(23)
cores = 8; T = 900                                                         # a 15 minute run, 1 Hz samples
t = np.arange(T); scene = 0.5 + 0.5 * np.sin(2 * np.pi * t / 240.0)         # scene complexity cycles: a common cause that correlates the loads
load = lambda mean, k, sd: np.clip(mean + k * (scene - 0.5) + sd * rng.standard_normal(T), 0.02, None)
cpu = {'vio': load(1.4, 0.5, 0.15), 'detector': load(0.7, 0.4, 0.15), 'mapper': load(1.1, 0.6, 0.25), 'planner': load(0.5, 0.6, 0.3), 'logging': load(0.4, 0.1, 0.08)}      # in cores
gpu = load(0.55, 0.3, 0.06)                                                 # fraction of the GPU
total = sum(cpu.values())
print(f"{'process':>9} {'mean':>6} {'p95':>6} {'max':>6}   (cores)")
for k, v in cpu.items(): print(f"{k:>9} {v.mean():6.2f} {np.percentile(v, 95):6.2f} {v.max():6.2f}")
print(f"{'combined':>9} {total.mean():6.2f} {np.percentile(total, 95):6.2f} {total.max():6.2f}   of {cores} cores;  GPU mean {gpu.mean():.2f}, p95 {np.percentile(gpu, 95):.2f}")
sum_p95 = sum(np.percentile(v, 95) for v in cpu.values())
print(f"sum of individual p95 = {sum_p95:.2f} cores vs combined p95 = {np.percentile(total, 95):.2f}  (these are different numbers: use the combined series)")
headroom = 1 - np.percentile(total, 95) / cores; need = 0.25
print(f"CPU headroom at the combined p95: {headroom:.0%} (required {need:.0%}) -> {'PASS' if headroom >= need else 'FAIL'}; GPU headroom {1 - np.percentile(gpu, 95):.0%}")
assert sum_p95 > np.percentile(total, 95) and headroom >= need and np.percentile(gpu, 95) < 0.8
Per-process and combined CPU load from time series, the difference between the sum of percentiles and the percentile of the sum, and a headroom check.

Experiments

  1. Record ten minutes of the real stack with the vehicle on the bench (sensors live), and then the same with a rosbag replay. Compare the combined p95: is the replay representative?
  2. Stress test: add a synthetic CPU load of one core on the cores used by the estimator and observe the effect on its timing (see the jitter lesson).
  3. Plot the combined load against time and mark the moments of the highest load. What is running then?
  4. Compute headroom for CPU, GPU and memory, write the budget row for each, and name the owner of each function.

How to know it worked

  • The analysis script runs and its assertions pass on the synthetic data.
  • You have per-process and combined p95 and peak for a real run of at least ten minutes.
  • You can state the headroom for CPU, GPU and memory and whether it meets the requirement.

If it does not work

SymptomLikely causeWhat to check
cpu_percent shows 0The counter was not primed or the interval is too shortCall cpu_percent(None) once before sampling and sample at 1 s
Sum of processes exceeds system CPUPer-process values are per coreDivide by the number of cores or report in cores
GPU shows 100 % constantlyThe tool reports the busy fraction of the time slice, not the capacity usedCompare against frame-time measurements before concluding saturation
Memory grows over the runA leak or a growing mapPlot resident set over time and run to the end of a full mission

References

Record your progress

Saved in this browser. This lab runs on your own machine, so the site cannot check it for you.