P9-T5 — Real-Time Constraints and Deployment

Learning objectives

After this lesson, you should be able to:

• Distinguish latency, throughput, jitter and tail latency, and say which one sets which requirement.
• Compute the compute utilisation of a model on a device and the memory of a model at different precisions.
• Explain why a design must use worst-case or high-percentile latency, not the mean.
• List the ways to fit a model to a device and the degradation strategies when a deadline is missed.

The idea: every result has a due date

A camera at 30 Hz delivers a frame every 33 ms. If the pipeline needs longer than that for a frame, either frames are dropped or the queue grows and everything is late. A result that arrives after the robot has moved is worse than useless if it is used as if it were current.

Four quantities

Why the tail matters

Latency depends on the scene. A cluttered scene produces many candidates for non-maximum suppression, a busy lidar sweep has more points to cluster, and a network that adapts its work to the number of objects takes longer when there are more. The slow frames come exactly when many things are happening. In addition, the computer has other jobs, caches are cold after a context switch, and a hot processor throttles its clock. The system must be designed for the slow frames, with the high percentile or the worst case, not the average.

Fitting a model to the device

1. A smaller model. Fewer layers or channels; a distilled copy of a big model; an architecture designed for the device.
2. Lower precision. Store and compute in 16-bit or 8-bit numbers instead of 32-bit: four times less memory and bandwidth at 8 bits, and faster on hardware that supports it, with a small accuracy loss that must be measured (quantisation).
3. Lower resolution or a region of interest. Cost scales with the number of pixels or tokens (Lessons P9-T1 and P9-T2). Run at lower resolution, or only on the part of the image that matters (where the tracker expects objects).
4. Skip frames or stages. Detect every third frame and track in between (Module 7), at the price of reacting later to new objects.
5. Move work off the critical path. Run slow, rich perception (a large model) asynchronously, and let a fast, simple path handle safety.

When the deadline is missed anyway

Plan for it. Drop the stale frame instead of processing it late, report the age of every output so that consumers can predict forward, switch to a cheaper model under load, slow the robot when the margin shrinks, and raise a health flag. A system that silently runs late is more dangerous than one that says so.

The mathematics: budgets and utilisation

Compute utilisation. A model that needs F floating-point operations per frame, run at f frames per second on a device that sustains P operations per second, uses the fraction

F

operations per frame (a MAC counts as two operations, Lesson P9-T1).

f

the required frame rate.

P

the sustained throughput of the device for this workload: the peak times an efficiency, often 20% to 50%.

Utilisation above 1 means the frame rate cannot be met; the system also needs room for the other stages, so a design target of 50% or less is typical.

Memory. A model with n parameters at b bytes each needs n · b bytes for its weights, plus activations. At 32, 16 and 8 bits, b = 4, 2 and 1.

Deadline misses. If the latency T of a frame has a distribution, the probability that a frame misses the deadline D is P(T > D). Over n frames, the expected number of late frames is n · P(T > D). The p-th percentile T_p is the latency below which a fraction p of the frames fall: a design to the 99th percentile misses the deadline on about 1% of the frames.

Pipelining. If the stages run concurrently on successive frames, the throughput is 1/max(t_i), but the latency is the sum of the t_i plus the waiting between stages (Lesson P1-T3). A queue between stages with a rate mismatch grows without bound.

Quantisation. Mapping a real value x in the range [x_min, x_max] to an 8-bit integer uses a scale s = (x_max − x_min)/255, so each value is stored to within s/2 of its true value. The loss in accuracy has to be measured on the real task.

Worked examples: budgets, tails and quantisation

Utilisation. A detector of 4 GFLOPs per frame at 30 Hz needs 120 GFLOP/s. A device with a peak of 1 TFLOP/s and an achievable 30% sustains 300 GFLOP/s: utilisation 120/300 = 0.4. A model of 20 GFLOPs would need 600 GFLOP/s, utilisation 2.0, so it can sustain only 15 Hz on this device.

Memory. A model with 25 million parameters: 25 × 10⁶ × 4 bytes = 100 MB in 32-bit, 50 MB in 16-bit, 25 MB in 8-bit. On a device with 512 MB shared between several models, the 8-bit version leaves room for the others and moves a quarter of the data per frame.

The tail. A pipeline has a mean latency of 31 ms and a 99th percentile of 80 ms; the deadline is 33 ms. The distribution in the figure has about 90% of frames near 28 ms and about 10% in a slow mode near 60 ms, so around 10% of frames miss the 33 ms deadline even though the mean (about 31 ms) is below it. At 30 Hz that is three late frames a second. A designer who looked only at the mean would have declared the system fine.

Reaction distance with the tail. A robot at 2 m/s with a mean latency of 31 ms travels 0.062 m before it can react; at the 99th percentile of 80 ms, 0.16 m. With braking at 1 m/s² and stopping distance 2.0 m, the latency adds 3% on average and 8% in the tail: modest here, but at 5 m/s the stopping distance is 12.5 m and the tail adds 0.4 m. The budget in Lab P9-L1 must use the high percentile.

Quantisation. A layer’s activations range from −6 to +6. An 8-bit mapping has s = 12/255 = 0.047, so each value is stored within 0.024 of the true one, about 0.2% of the range. If the range is dominated by a rare outlier of 60, the same 8 bits give s = 0.47: ordinary values are now stored to ±0.24, 4% of their own range, and accuracy falls. The range choice (calibration) matters as much as the bit width.

Engineering implications and limitations

Measure on the target. Latency and throughput on a workstation tell you little. Time every stage on the robot’s computer, with the other processes running, over a long, varied log. Report the mean, the 99th percentile and the maximum.

Design to the tail. Use a high percentile or the maximum for the budgets. Use a real-time scheduler, fixed priorities and pinned cores for the critical path, and avoid memory allocation and garbage collection inside it.

Timestamp at capture. Pass the capture time through the pipeline so that the consumer knows the age of every result, and can predict it forward (Lesson P8-T3).

Drop, do not queue. If a stage is late, discard the oldest frame and take the newest. A growing queue turns a small overload into a permanent delay.

Check accuracy after every optimisation. Quantisation, pruning, lower resolution and frame skipping each change the behaviour. Re-run the evaluation by condition (Lesson P6-T6), not just the headline metric.

Watch temperature and power. A processor that throttles at 80 °C may run at half speed in a hot enclosure in summer. Test in the worst environment and add a thermal margin.

Define the fallback. If perception is late or missing, the robot slows, stops or falls back to a simpler sensor path. Test it by injecting delays (Lab P9-L1, Lesson P9-T6).

Common misconceptions

Engineering takeaways

1. Latency (age), throughput (rate), jitter and tail latency are different; the robot’s reaction distance depends on latency, its update rate on throughput.
2. Utilisation = F·f/P: 4 GFLOPs at 30 Hz needs 120 GFLOP/s, 40% of a device that sustains 300; 20 GFLOPs would not fit.
3. Memory scales with bytes per parameter: 25 M parameters is 100, 50 and 25 MB at 32, 16 and 8 bits.
4. Design to a high percentile or the worst case: a mean of 31 ms can hide 10% of frames over a 33 ms deadline.
5. Fit the model with smaller models, lower precision, lower resolution, regions of interest and skipped frames, re-check accuracy, and plan for missed deadlines.

References and further reading

Papers and a textbook on tail latency, quantisation, performance models and real-time systems. The model sizes, device throughput and latency distribution in the examples are illustrative.

• Dean, J. and Barroso, L. A. (2013). The Tail at Scale. Communications of the ACM, 56(2).
• Jacob, B. et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR.
• Williams, S., Waterman, A. and Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4).
• Buttazzo, G. C. (2011). Hard Real-Time Computing Systems, 3rd ed. Springer: deadlines and worst-case execution time.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: how to test a perception system for robustness, find its failures and prove less than you think.

Related content

Nothing below is required to finish this lesson.

• Before
  P9-T4: Perception for Manipulation and Autonomous Navigation

  State what manipulation and navigation each need from perception.

• Lab
  P9-L1: Evaluate Robustness and Latency Budgets

  Measure how a pipeline degrades under injected faults and whether it meets a latency budget.