Allocating workloads to CPU and GPU

1. Engineering problem

Moving a stage to the GPU can speed it up tenfold or slow it down. The deciding factors are how parallel the work is, how many bytes must cross to the device and back, and whether the stages around it can keep the GPU fed.

2. When is this needed?

• A stage dominates the pipeline (see the profiling guide).
• Adding neural-network perception.
• GPU utilisation is low or CPU is saturated.

3. Selection and design decisions

Transfer and parallelism

• Transfer: offload wins only if copy + kernel + launch overhead is below the CPU time. Avoid ping-ponging data between CPU and GPU between stages.
• Amdahl: if a fraction p of a stage speeds up by s, the whole stage speeds up by 1 / ((1 − p) + p/s). A 10× faster 90 % still gives only about 5×.
• Shared memory SoCs: CPU and GPU share bandwidth; a busy GPU can slow the CPU.
• Batching: processing several frames at once raises GPU efficiency and adds latency.

4. Advantages and disadvantages

• GPU offload cuts CPU load and raises power and complexity (drivers, CUDA versions, debugging).
• Keeping data on the GPU between stages saves copies but couples the stages.
• More processes isolate failures and use more cores, but add copies between them.

5. Requirements and constraints

• Memory on the GPU for models and buffers.
• Software support: CUDA toolkit and library versions available on the platform.
• Latency budget: GPU queueing and batching add delay.
• Process boundaries: large messages between processes are copied (see intra-process communication in Module 17).

6. Architecture

camera ─▶ [CPU: driver, decode] ─▶ [GPU: preprocess → network → postprocess] ─▶ [CPU: tracking, fusion] ─▶ planner
LiDAR  ─▶ [CPU: deskew, filter] ─▶ [CPU: odometry / SLAM] ───────────────────────────────────────────────▶ localisation

7. Implementation

def offload_wins(cpu_ms, gpu_ms, bytes_moved, link_bytes_per_s, overhead_ms=0.05):
    """GPU path = copy to the device + kernel + launch overhead (copy back assumed small)."""
    transfer_ms = bytes_moved / link_bytes_per_s * 1000
    gpu_path = transfer_ms + gpu_ms + overhead_ms
    return round(gpu_path, 2), gpu_path < cpu_ms

frame = 1280 * 720 * 3                                  # one rgb8 frame, bytes
link = 4e9                                              # assumed effective host-to-device rate, bytes/s (an assumption, not a product figure)
print(offload_wins(40.0, 6.0, frame, link))             # heavy per-frame work: offload
print(offload_wins(0.3, 0.1, frame, link))              # tiny work: the copy costs more than the saving
p, s = 0.9, 10.0                                        # 90 % of the stage is parallel and runs 10x faster
print(round(1 / ((1 - p) + p / s), 2))                  # Amdahl: the serial 10 % limits the gain

Printed (checked by the site’s tests):

(6.74, True)
(0.84, False)
5.26

Reading it: for a 40 ms CPU stage on a 2.76 MB frame, the GPU path is 6.74 ms including the copy, so offload clearly wins; for a 0.3 ms CPU task, the same copy makes the GPU path 0.84 ms and the CPU wins. Amdahl: a stage that is 90 % parallel and runs 10× faster only speeds up 5.26× as a whole.

Distributing ROS 2 nodes

Not executed by the site’s tests:

htop                                           # per-core load; F5 shows the process tree
taskset -c 0-3 ros2 run my_pkg my_node         # pin a process to cores 0-3 (Linux)
ps -eLo pid,tid,pcpu,comm --sort=-pcpu | head  # busiest threads
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1   # NVIDIA GPU use per second

Pin only to fix a measured contention problem; the scheduler usually does better. Compose nodes that exchange large messages into one process to avoid copies.

8. Real examples

• Detector on the GPU, SLAM on the CPU: the common split; keep the image on the GPU through preprocessing and inference.
• Small point cloud filtered on the GPU: slower than the CPU because of the copy.
• GPU 30 % busy, CPU core at 100 %: the CPU stage before the GPU starves it (see the diagnostic).

9. Failure scenarios

• Offloading tiny kernels and paying more in transfer than the saving.
• Copying data to the GPU and back at every stage.
• Ignoring shared memory bandwidth on an SoC.
• Expecting the whole pipeline to speed up by the kernel’s speed-up.

10. Troubleshooting

1. Profile first (profiling guide) and find the dominant stage.
2. Measure its CPU time and the bytes it processes.
3. Compute the break-even; prototype only if it is plausibly positive.
4. After the change, re-measure end-to-end latency, not just the kernel.

11. Validation

• End-to-end latency and frame rate improve at full load, not only the stage time.
• CPU load and GPU busy fraction are inside the margins (CPU ≤ 70 %, GPU ≤ 80 %).
• No thermal throttling in a one-hour run.
• The break-even example prints the values above (executed by the site tests).

12. Engineering checklist

• Dominant stage identified by measurement
• Transfer cost estimated
• Amdahl limit considered
• Data kept on the GPU between GPU stages
• Shared memory bandwidth considered
• Software and driver versions supported
• End-to-end latency re-measured

13. Further references

• CUDA and GPU programming guides from the GPU vendor for transfer, streams and unified memory (not fetched here).
• Amdahl’s law: standard computer-architecture texts.

Related on FixTheRobot

• Real-Time Performance and Resource Constraints SLAM · Lesson