M9-T2 — Engineering Tradeoffs and System Evaluation

Learning objectives

After this lesson, you should be able to:

• Identify the Pareto-optimal designs among candidates and explain why a weighted score can mislead.
• Perform a sensitivity analysis on decision weights and on a design parameter.
• Use the rule of three to compute how many failure-free tests support a reliability claim.
• Write a verification plan and an engineering report with requirement, method, result, limits and untested conditions.

The problem and the intuition

Suppose four candidate systems are compared on error and CPU time. If one is both worse in error and costs more CPU than another, it is simply dominated: no requirement would choose it. What remains is a front of designs where improving one measure costs the other. Picking one of them is not a calculation but a decision, and the decision belongs to the requirements: a battery-powered robot picks the cheap end, a survey system the accurate end.

A common mistake is to hide this decision inside a weighted score. A number that combines accuracy, robustness, compute and cost depends on the weights, and the weights are opinions; two reasonable people with slightly different weights can choose different winners. The honest report shows the measurements, the front and the sensitivity of the choice to the weights.

The second theme is evidence. “It worked in all our tests” means something only if the number of tests is stated. If a system failed 0 times in 30 missions, its true failure rate could still be 10%. Reliability claims need numbers of trials that match the claim.

Report measurements, show the Pareto front, test the sensitivity of the choice, and state how many trials support each reliability claim.

Tradeoffs, sensitivity and evidence

Problem. Choose among candidate SLAM designs and justify the choice, with evidence of an appropriate strength, and communicate it in a form that others can check.

Assumptions. Candidates can be measured on common data with a fixed protocol (M8-T8); the requirements give limits on each measure; trials are independent enough for simple statistics.

Pareto optimality

Design X dominates design Y if X is at least as good as Y on every measure and strictly better on at least one. The non-dominated designs form the Pareto front. Eliminate dominated designs first; choose among the rest by the requirements (hard limits, then preferences).

Weighted decision matrices and their sensitivity

Score each design 0–10 on each criterion, weight the criteria (weights sum to 1), and add. It is a way of organising a discussion, not a measurement. Always repeat the calculation with perturbed weights: if the winner changes, the decision is not supported by the scores.

Sensitivity and ablation

• Parameter sensitivity: vary one parameter at a time (voxel size, thresholds, window) and plot the metric. A system whose accuracy collapses with a 20% change in a threshold is fragile.
• Ablation: remove one component (loop verification, gating, IMU) and measure the loss. This shows which parts earn their cost and where the risks are (M8-L4).
• Fault injection: add controlled noise, outliers, dropouts and delays and plot degradation curves, with failure rate (M8-T8).

How much testing supports a reliability claim

If n independent trials all succeed, the largest failure probability p still consistent with that at confidence 1 − α satisfies (1 − p)ⁿ = α. For 95% confidence (α = 0.05) this gives the rule of three (Hanley and Lippman-Hand, 1983):

To support a failure rate below 1 in 100 with zero observed failures, about 300 trials are needed; below 1 in 1 000, about 3 000; below 1 in 10 000, about 30 000. If failures occur, use exact binomial intervals. Trials must cover the conditions of the claim (lighting, traffic, route); trials in one benign condition support a claim only about that condition.

Verification plan

The engineering report

• Requirements as numbers, with their source.
• Design (architecture, sensors, parameters, versions) in enough detail to reproduce.
• Method (data, alignment, metrics, number of runs, how failures are counted).
• Results with median, spread and failure rate; the Pareto front; sensitivity and ablations.
• Failure analysis: what failed, the cause, the evidence, the fix, and what was not fixed.
• Limits and untested conditions, stated plainly; the claims supported by the evidence and no more.

Pareto front

the set of designs not dominated by any other.

ablation

removing a component to measure its contribution.

rule of three

upper 95% bound 3/n on a failure probability after n failure-free trials.

acceptance criterion

the pre-agreed threshold on a metric that decides pass or fail.

A worked example: a tie, a flip, and a reliability number

1. A decision matrix. Criteria and weights: accuracy 0.4, robustness 0.3, compute 0.2, cost 0.1. Scores (0–10): system A (2D lidar + SLAM Toolbox): 8, 6, 9, 5. System B (lidar + IMU + wheels with verified loop closure): 6, 9, 7, 8. System C (stereo VIO): 9, 5, 5, 9. Totals: A = 0.4 × 8 + 0.3 × 6 + 0.2 × 9 + 0.1 × 5 = 7.3; B = 2.4 + 2.7 + 1.4 + 0.8 = 7.3; C = 3.6 + 1.5 + 1.0 + 0.9 = 7.0. A and B tie.
2. The sensitivity check. Move 0.1 of weight from accuracy to robustness (0.3, 0.4, 0.2, 0.1): A = 2.4 + 2.4 + 1.8 + 0.5 = 7.1; B = 1.8 + 3.6 + 1.4 + 0.8 = 7.6. B wins clearly. Move it the other way (0.5, 0.2, 0.2, 0.1): A = 4.0 + 1.2 + 1.8 + 0.5 = 7.5; B = 3.0 + 1.8 + 1.4 + 0.8 = 7.0. A wins. The winner depends on a 0.1 change in a weight: the scores do not decide; the requirements do. If robustness matters more to this customer, B.
3. Dominance. Suppose measured: A has ATE 0.12 m and 0.6 core; D has ATE 0.15 m and 0.8 core. D is dominated by A (worse in both) and can be dropped without any weights.
4. A reliability claim. The team ran 30 missions without a localisation failure. By the rule of three, the 95% upper bound on the failure probability is 3 / 30 = 10%: consistent with one failure in ten. To claim fewer than one in 1 000 they need about 3 000 failure-free missions. Equivalently, if the true rate were 1%, the chance of seeing no failure in 30 missions is 0.99³⁰ = 74%: 30 clean runs say almost nothing about a 1% rate.
5. Reading a result honestly. “Median ATE 0.12 m over 20 runs, 1 failure (ATE 3.1 m, false loop accepted in run 14), failure rate 5% (95% interval 0.1% to 25%)” is a complete statement. “0.12 m average error” is not.

Engineering implications and limitations

Engineering implications

• Fix the protocol before the results. Decide metrics, runs and acceptance thresholds in advance, so that results cannot be tuned to the story.
• Show the front and the sensitivity, not a single winning score.
• Budget the testing for the reliability claim, and design the test campaign to cover the claimed conditions.
• Keep the failures. They are the most valuable output of testing; analyse each one and keep the logs.
• Version everything: data, code, parameters and models, so that the report can be reproduced.
• Evaluate with the tools the field uses, and write down how. Run evo_ape and evo_rpe (or rpg_trajectory_evaluation) on trajectories saved in a standard format, with the alignment that fits the sensor (Sim(3) for monocular, SE(3) for stereo, RGB-D and LiDAR, 4 degrees of freedom for visual-inertial), and put the exact command, tool version and time-association limit in the report. If you also publish on a public benchmark, use its own trajectory format and metric (KITTI odometry error, TUM and EuRoC ATE and RPE; M8-T8) so the numbers can be compared with published ones.
• State what was not tested. Readers need to know the boundary of the evidence.

Limits

Pareto analysis ignores the shape of the distributions (a design with a good median and a catastrophic tail can sit on the front); decision matrices encode judgement; the rule of three assumes independent, identically distributed trials, and real missions are correlated (the same route, the same day). Exact binomial intervals are better when failures occur, and statistical claims about rare events need either very many trials or modelling assumptions that should be stated. Safety-critical applications need formal safety cases that go beyond this lesson.

Common misconceptions

Engineering takeaways

1. Eliminate dominated designs, then choose along the Pareto front using the requirements.
2. A weighted score is a discussion aid: check its sensitivity to the weights.
3. Rule of three: zero failures in n trials gives an upper bound of about 3/n at 95% confidence; a 1-in-1 000 claim needs about 3 000 trials.
4. Verification plans link each requirement to a test, a metric, a protocol and an acceptance threshold.
5. An engineering report states requirements, design, method, results with spread and failures, failure analysis, and limits.

References and further reading

Primary papers and authoritative documentation.

• Hanley, J. A. and Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 249(13).
• Sturm, J., Engelhard, N., Endres, F., Burgard, W. and Cremers, D. (2012). A benchmark for the evaluation of RGB-D SLAM systems. IEEE/RSJ IROS.
• Zhang, Z. and Scaramuzza, D. (2018). A tutorial on quantitative trajectory evaluation for visual(-inertial) odometry. IEEE/RSJ IROS.
• Cadena, C. et al. (2016). Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics 32(6).
• Brown, L. D., Cai, T. T. and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science 16(2).
• Deb, K. (2001). Multi-Objective Optimization using Evolutionary Algorithms. Wiley (Pareto optimality).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The capstone lab puts the pipeline together: a front end that fuses and gates, a loop verifier, and an end-to-end run with injected failures to diagnose and report.

This is the last lesson of the track.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T1: End-to-End SLAM System Architecture

  Design a complete SLAM architecture from sensor choice to evaluation and deployment.

• Lab
  M9-L1: Build and Debug an End-to-End SLAM Pipeline

  Assemble odometry, scan matching, pose graph, mapping and evaluation, diagnose injected failures and write an engineering report.