P9-T6 — Robustness, Evaluation and Failure Analysis

Learning objectives

After this lesson, you should be able to:

• Name the main sources of perception failure and place them in the pipeline.
• Design a fault-injection and slice-based evaluation, and report a worst case and a break point.
• Describe runtime monitoring and the limits of out-of-distribution detection.
• Compute the upper bound on a failure rate from a number of failure-free trials and the number of trials needed for a target.

The idea: look for the failure, do not wait for it

Perception fails in ways that follow the pipeline. Using the stages of Lesson P1-T3, here is where.

Distribution shift and corruptions

Every learned or tuned component was fitted on conditions that the deployment will not exactly reproduce (Lesson P6-T6; the VLA track treats shift for policies). Benchmarks of corruptions apply blur, noise, weather, compression and brightness changes to a test set at several severities, and measure how fast performance falls. They show that accuracy on the clean test set is a poor predictor of accuracy under such changes, and that models differ greatly in how they degrade.

Fault injection

Instead of waiting for faults, create them. Drop frames, delay a sensor, rotate a camera by a degree, add noise, black out a quadrant, remove a sensor entirely, feed a lidar scan with a shadow. For each fault, at several severities, measure the system’s outcome (not just a component’s): does the robot still stop in time, does the report stay within tolerance, does the system notice? Lab P9-L1 does this for the detector and Lab P9-L2 for the diagnosis of calibration faults. The result is a table of break points: the severity at which the system fails, per fault.

Slices and worst cases

Report performance by slice (condition, range, size, class, occlusion) and give the worst slice with its sample size (Lesson P6-T4). An average over a mix hides the case that matters. A slice with few examples has a wide interval; the right response is to collect more examples of it.

Monitoring in operation

No test covers everything, so the system should watch itself. Useful checks need no ground truth: the age of every input, the agreement between sensors (Lesson P8-T5), the score distribution and the number of detections per frame compared with their usual values, the fraction of the image that is saturated or dark, the conflict rate, and the track birth and death rates (Lesson P7-T5). Out-of-distribution detectors flag inputs unlike the training data; they help, but they miss shifts that look normal to the detector and flag harmless ones, so they are one signal among several. A monitor should lead to an action: slow down, widen margins, ask for help or stop.

Analysing a failure

1. Log the raw inputs and every interface (Lesson P1-T3), with timestamps. A failure that cannot be replayed cannot be analysed.
2. Find and classify the failure: miss, false alarm, wrong position, wrong identity, late, or no output.
3. Replay it through the pipeline and examine each interface in order, from the first. The first stage whose output is wrong is where to look; later stages inherited the error.
4. Name the cause: data (a condition not covered), model (a systematic error), alignment (calibration, time), algorithm (a gate too tight), system (a deadline missed).
5. Fix it, and add the case to the test set with the fault injections that reproduce it, so that it cannot silently return.

The mathematics: what failure-free trials prove

Suppose you run n independent trials and observe no failures. What can you say about the true failure probability p? The chance of seeing none is (1 − p)ⁿ. The largest p that is still compatible with the observation at confidence 1 − α satisfies (1 − p)ⁿ = α, so

n

the number of independent trials with zero failures.

α

the allowed chance of being wrong; 0.05 gives 95% confidence.

p_upper

the failure probability you cannot rule out at that confidence. For α = 0.05, −ln 0.05 = 3.0: the “rule of three”.

Inverting the rule: to show a failure probability below p_target at 95% confidence you need about 3 / p_target failure-free trials.

With failures. If k failures occur in n trials the estimate is k/n with an interval that is wide when k is small (Lesson P6-T4); the rule of three is the special case k = 0.

Independence. The bound assumes the trials are independent and representative of deployment. Fifty frames from one drive are not fifty independent trials: they share a scene. The effective n is closer to the number of distinct situations.

Fault-injection break point. For a fault with severities s₀ < s₁ < … and an outcome measure m(s) (recall, error, margin), the break point is the first s_i at which m crosses the acceptance limit: for recall, the first severity with recall below r times its clean value (Lab P9-L1, stage 3).

Worked examples: how much a clean run proves

The rule of three. A perception system detects every one of 300 test obstacles. Zero failures in 300 trials: the 95% upper bound on the failure probability is 3/300 = 1%. The system may well be better than that, but 300 clean trials do not show it. To demonstrate a failure probability below 0.1% you need about 3 / 0.001 = 3,000 failure-free trials; below 0.001% (10⁻⁵), 300,000. The exact bound for 300 trials is 1 − 0.05^1/300 = 0.99%.

Correlated trials. Those 300 obstacles came from 6 test drives of 50 frames each. The frames of one drive share lighting and background, so they are not independent: the effective number of trials is nearer 6 to 30 situations, and the bound is 3/6 = 50% to 3/30 = 10%, not 1%. Counting frames overstates what the test proves by an order of magnitude.

A fault-injection table. Recall of a detector under three faults at four severities, as in Lab P9-L1 (illustrative values):

With an acceptance limit of 0.9 × the clean recall (0.68): dropout breaks at step 1 (0.61), darkness at step 2 (0.67), box noise at step 3 (0.32, after holding at 0.75). Box noise looks harmless until it falls off a cliff: the IoU threshold makes the loss abrupt. Dropout degrades steadily. The worst case at the highest severity is box noise (0.32), the lowest of the three. A single “average recall” of 0.55 over all the entries would hide the cliff and the early break of dropout.

A diagnosis. A robot clips a pallet at a distance of 12 m. The log shows: the detector found it (score 0.84); the lifted position was 0.42 m to the right of the lidar’s; the residual for other objects grows with range. The first stage whose output is wrong is the alignment, with a pattern of a rotation error: 0.42/12 = 0.035 rad = 2.0°. The fix is to recalibrate the camera yaw, and the added test injects a 2° yaw error at several ranges, checks that the monitor flags it (Lab P9-L2, stage 3), and checks that the robot stops in time with the error present.

Engineering implications and limitations

Build the fault injector early. A tool that replays logged data with faults (dropped frames, a delay, a rotated camera, noise, a missing sensor) is the most useful test asset. Run it in CI on every change.

Write acceptance limits before the experiment. Decide what outcome counts as failing (a margin, a recall) and report the break point per fault, with the worst slice and the number of independent examples behind it.

Count independent situations, not frames. Group frames by scene and report how many distinct situations a claim rests on. Use the rule of three on those.

Separate what you tested from what you assume. A safety statement should list the conditions covered by tests, the conditions assumed, and the monitors that detect leaving the covered range.

Monitor and act. Every monitor needs a defined response, tested by injection: if conflicts exceed a rate, slow down; if an input is older than a limit, stop using it; if the score distribution shifts, widen margins.

Keep every failure. Archive the raw log of each field failure and each near miss, with the diagnosis, and add them to the regression set. The set grows with the system’s experience.

Do not oversell. A passed test suite shows that these tests pass. State the limits of the evidence, as this track has stated them for each lab: simulated data, stated error models and no claim of real-world safety.

Common misconceptions

Engineering takeaways

1. Failures follow the pipeline: sensing, alignment, detection, tracking, fusion, representation and system timing each have typical faults.
2. Test by injecting faults at several severities and report break points per fault, the worst slice and the sample size behind it.
3. Monitor without ground truth (input age, sensor agreement, score and detection statistics, conflict and track rates) and attach a tested action to every monitor.
4. Rule of three: zero failures in n independent trials bounds the failure probability at about 3/n (1% for 300 trials; 3,000 trials for 0.1%); correlated frames reduce the effective n.
5. Analyse every failure by replaying it, locating the first wrong interface, naming the cause, and adding it to the test set.

References and further reading

Papers and reports on robustness benchmarks, failure-free test bounds, uncertainty under shift and autonomous system validation. The recall table is illustrative, of the form produced by Lab P9-L1; the calculations were checked.

• Hendrycks, D. and Dietterich, T. (2019). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR.
• Hanley, J. A. and Lippman-Hand, A. (1983). If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators. JAMA, 249(13).
• Ovadia, Y. et al. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS.
• Amodei, D. et al. (2016). Concrete Problems in AI Safety. arXiv:1606.06565.
• Koopman, P. and Wagner, M. (2016). Challenges in Autonomous Vehicle Testing and Validation. SAE International Journal of Transportation Safety.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes the lessons. The capstone, Lab P9-L2, builds an end-to-end pipeline and applies them.

This is the last lesson of the track.

Related content

Nothing below is required to finish this lesson.

• Before
  P9-T5: Real-Time Constraints and Deployment

  Budget latency, compute and memory for a perception pipeline.

• Lab
  P9-L1: Evaluate Robustness and Latency Budgets

  Measure how a pipeline degrades under injected faults and whether it meets a latency budget.

• Lab
  P9-L2: Capstone: Build an End-to-End Perception Pipeline

  From simulated camera and range observations to a structured obstacle report for a downstream task, with diagnosis of injected failures.