GPU utilization is unexpectedly low

1. Symptoms

• nvidia-smi (or tegrastats on Jetson) shows low GPU use while the detector or other GPU stage is slower than required.
• One CPU core is at 100 % while the GPU is idle.
• Throughput does not improve with a faster GPU.
• GPU use is high only in short bursts.

2. Possible causes

• CPU-side bottleneck before the GPU: decoding, resizing, copying or Python pre-processing in a single thread.
• Transfer overhead: many small copies between host and device; unpinned memory.
• Small batches or tiny kernels: the GPU finishes instantly and waits.
• Synchronisation: the CPU waits for each result before submitting more work.
• Input starvation: the source delivers frames slowly (camera rate, network, QoS drops).
• Power mode or throttling lowering GPU clocks, which also hides true utilisation.
• Model on the CPU: the framework fell back to the CPU for the model or some operators.

3. Measurements to collect

• GPU utilisation and memory use over time (per second).
• CPU use per core and per thread of the process feeding the GPU.
• Input frame rate versus processed frame rate.
• Per-stage time: preprocessing, copy, inference, postprocessing.
• GPU clocks, power mode and temperature.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

nvidia-smi dmon                                          # per-second GPU utilisation, memory, clocks (NVIDIA GPUs)
nvidia-smi --query-gpu=utilization.gpu,memory.used,clocks.sm --format=csv -l 1
nvidia-smi -q -d PERFORMANCE                             # clock throttle reasons
tegrastats                                               # Jetson only: GPU, CPU, memory, temperature, power
htop                                                     # is one core pegged?
top -H -p <PID>                                          # which thread of the process
ros2 topic hz /camera/image_raw                          # input rate
ros2 topic hz /detections                                # output rate

5. Step-by-step investigation

1. Confirm the GPU is really the stage in question: compare input and output rates; if the output rate equals a low input rate, the GPU is not the limit.
2. Check the feeding thread. If a CPU core is saturated in the process that feeds the GPU, the CPU is the limit: profile preprocessing (resize, colour conversion, copies).
3. Time each stage (preprocess, copy, kernel, postprocess). If the kernel is a small share, optimise the others.
4. Check transfers. Many small copies or pageable memory slow transfers; batch them or use pinned memory where your framework allows.
5. Check batching and synchronisation. Submit work asynchronously and overlap copy and compute where the framework supports it; avoid waiting for every result.
6. Check clocks and power mode. A lower power mode or thermal throttling shows low clocks; fix cooling or the mode first.
7. Confirm the model really runs on the GPU (framework device logs) and that unsupported operators do not fall back to the CPU.

6. Corrective actions

• Move preprocessing to the GPU or parallelise it across CPU cores.
• Keep data on the GPU between stages; avoid round trips.
• Batch frames where the latency budget allows.
• Make submission asynchronous and overlap transfer and compute.
• Raise the power mode or fix cooling if clocks are low.
• Replace operators that fall back to the CPU, or change the export/optimisation of the model.
• Fix the input rate if the source is the limit.

7. Validation after the fix

• End-to-end frame rate reaches the requirement at full load.
• GPU busy fraction is in your design range (not necessarily 100 %: leave headroom).
• No CPU core is saturated in the feeding process.
• The result holds after an hour at full load with stable clocks.

References

• See the workload-allocation and profiling guides for the break-even and measurement routine; GPU vendor tools documentation for deeper profilers (not fetched here).

• Real-Time Performance and Resource Constraints SLAM · Lesson