Measuring end-to-end latency

1. Engineering problem

A robot reacts to the world after sensing, transport, perception, planning and control. The delay from the physical event to the motor command is the end-to-end latency; it sets how fast the robot can safely move. Averages hide the rare long delays that cause collisions.

2. When is this needed?

• Setting speed limits from the reaction time.
• When the robot reacts late.
• After any pipeline change.

3. Selection and design decisions

ROS 2 provides ros2 topic delay, hz and bw for per-topic measurements, which depend on header stamps being acquisition times; the bag-latency guide in this track shows how to measure stamp-to-receive offsets from a recording. For an end-to-end figure, compare the stamp of the original sensor message with the time the last stage’s output (for example the velocity command) is produced, carrying the original stamp through the pipeline, and use a clock common to all machines.

4. Advantages and disadvantages

• More instrumentation gives detail and adds overhead.
• Carrying the original stamp through the pipeline gives true end-to-end numbers and needs every node to preserve it.
• Percentiles need many samples to be meaningful in the tail.

5. Requirements and constraints

• A common clock across machines (see the synchronisation guides).
• Original stamps are preserved by each stage.
• The budget is derived from speed and stopping distance, and applied to a high percentile or the maximum.

6. Architecture

event ─▶ sensor ─▶ driver ─▶ transport ─▶ perception ─▶ planning ─▶ command ─▶ actuator ─▶ motion
   t0 (stamp)                                                        t1 (command out)   latency = t1 − t0

7. Implementation

1. Choose the path to measure (sensor to command).
2. Preserve the sensor stamp through each stage and log the output time.
3. Record a long run under full load.
4. Compute percentiles and per-stage breakdown (below).
5. Compare the tail with the budget and fix the largest contributor.

def pct(xs, p):
    s = sorted(xs); k = (len(s) - 1) * p / 100
    f = int(k); c = min(f + 1, len(s) - 1)
    return s[f] + (s[c] - s[f]) * (k - f)
stages = {"sensor+driver": [8, 9, 8, 10, 9, 8, 30, 9, 9, 8], "transport": [2, 2, 3, 2, 2, 2, 2, 25, 2, 2], "perception": [20, 21, 20, 22, 21, 20, 21, 20, 60, 21], "planning": [10, 10, 11, 10, 10, 10, 10, 10, 10, 10]}
n = 10
total = [sum(stages[s][i] for s in stages) for i in range(n)]
print("end-to-end per frame (ms):", total)
print(f"p50 {pct(total, 50):.1f}, p95 {pct(total, 95):.1f}, max {max(total)} ms")
for s, v in stages.items():
    print(f"  {s:14s} median {pct(v, 50):5.1f} ms, worst {max(v)} ms")
budget = 60
over = sum(1 for t in total if t > budget)
print(f"budget {budget} ms exceeded in {over} of {n} frames ({over / n * 100:.0f} %): judge the tail, not the mean (mean {sum(total) / n:.1f} ms)")

Printed (checked by the site’s tests):

end-to-end per frame (ms): [40, 42, 42, 44, 42, 40, 63, 64, 81, 41]
p50 42.0, p95 73.4, max 81 ms
  sensor+driver  median   9.0 ms, worst 30 ms
  transport      median   2.0 ms, worst 25 ms
  perception     median  21.0 ms, worst 60 ms
  planning       median  10.0 ms, worst 11 ms
budget 60 ms exceeded in 3 of 10 frames (30 %): judge the tail, not the mean (mean 49.9 ms)

Reading it: ten frames give a median of 42 ms but a 95th percentile of 73.4 ms and a worst case of 81 ms; the per-stage worst cases (30 ms in the driver, 25 ms in transport, 60 ms in perception) show where the tail comes from. Against a 60 ms budget, 30 % of frames fail even though the mean is 49.9 ms. The data are illustrative.

8. Real examples

• AMR: a 60 ms sensor-to-command budget derived from speed and stopping distance, checked at the 99th percentile over a one-hour run.

9. Failure scenarios

• Reporting the mean latency.
• A camera driver that stamps at arrival.
• Measuring at idle, not under full load.

10. Troubleshooting

1. Break the tail down by stage.
2. Check clock sync between machines.
3. Look for correlated spikes (garbage collection, another process, Wi-Fi).

11. Validation

• The example prints the lines shown (executed by the site tests).
• The measured tail under full load meets the budget with margin.

12. Engineering checklist

• Common clock
• Stamps preserved
• Percentiles reported
• Per-stage breakdown
• Budget from stopping distance
• Measured under full load

13. Further references

• ROS 2 command-line tools ros2 topic delay, hz and bw (Jazzy); see the bag latency guide.

Related on FixTheRobot