P1-T6 — How to Evaluate a Perception System

Learning objectives

After this lesson, you should be able to:

• Choose an evaluation metric that matches a stated robot task and its costs.
• Compute precision, recall, accuracy and F1 from a confusion matrix, and explain the base-rate problem.
• Summarise a position error by its distribution (median, percentiles, worst case) instead of the mean only.
• Identify evaluations that mislead: leakage, unrepresentative data, averaging over easy cases and untested conditions.

The idea: ask what the robot needs, then measure that

A fire alarm that never sounds is 100% accurate on the thousands of days with no fire. It is also useless. Accuracy counts all correct decisions equally; a fire alarm must be judged on the rare days that matter. The same is true of an obstacle detector: most frames contain no obstacle in the stopping path, so a detector that always says “clear” is right most of the time.

Evaluation starts with a question about the task, not about the data:

1. What does the robot do with the output? Stop? Steer? Grasp? Count items?
2. What does each kind of error cost? A missed pedestrian and a false stop have very different costs. A grasp that is 1 cm off might be fine for a bottle and fatal for a pin.
3. Which conditions will it meet? Day and night, rain, clutter, far and near, the sensor on a dusty day. Evaluate in each, not on average.
4. Which metric answers the question? Recall for “does it find obstacles?”, position error percentiles for “is it close enough to grasp?”, track continuity for “does it keep the same person?”

A metric is a decision about what matters. Report the one that matches the task, and show the others next to it.

The mathematics: counting successes and failures

For a yes/no question (is there an obstacle?), every decision falls into one of four cells of the confusion matrix: TP (true positive: said yes, was yes), FP (false positive: said yes, was no), FN (false negative: said no, was yes) and TN (true negative: said no, was no). From them:

precision

of the things the system reported, the fraction that were real. Low precision means false alarms.

recall

of the real things, the fraction the system reported. Low recall means misses. Also called sensitivity or detection rate.

accuracy

the fraction of all decisions that were right. Dominated by TN when positives are rare.

F1

the harmonic mean of precision and recall: high only when both are high. It treats misses and false alarms as equally costly, which is rarely true.

Precision and recall trade against each other through the decision threshold (Lesson P6-T5 develops this): a more cautious detector gains recall and loses precision.

The base-rate problem

Let the fraction of frames with an obstacle be π (the base rate). A system that always says “no obstacle” has accuracy 1 − π, which is 0.97 when π = 0.03, with recall 0. Accuracy rewards ignoring the rare event. When positives are rare, report precision and recall, and the number of positives in the test set so that the reader can judge the uncertainty.

Continuous errors: look at the distribution

For position estimates, compute the error of each estimate against the ground truth e_i = ‖x̂_i − x_i‖ and summarise it by

The RMSE penalises large errors heavily; the median ignores them; the 95th percentile (p95) says the error is below this value in 95% of cases; the maximum is the worst case seen. For a safety question the tail matters more than the middle: two systems with the same median error can differ ten times in p95.

Uncertainty about the evaluation itself

A test set of N samples gives a measured rate with a statistical uncertainty. For a measured recall p̂ over n true objects, the standard error is about

With 30 true obstacles and a measured recall of 0.60, SE = √(0.6 × 0.4 / 30) ≈ 0.09. So the true recall could easily be anywhere from about 0.42 to 0.78. A recall measured on 30 events is a rough guide, not a number to three decimals.

A worked example: two detectors, same accuracy

An obstacle detector is tested on 1 000 frames; 30 contain an obstacle in the stopping path. The detector’s results are in the figure above: TP = 18, FN = 12, FP = 6, TN = 964.

The accuracy looks excellent. But 12 of 30 obstacles were missed. A second detector is more cautious: TP = 28, FN = 2, FP = 40, TN = 930.

Its accuracy is lower (0.958 against 0.982), yet it misses only 2 obstacles. Which is better depends on the costs. If a miss means a collision and a false alarm means a one-second pause, the second detector is clearly preferable: 40 pauses are far cheaper than 10 more collisions. If the robot is a vacuum cleaner and false stops are annoying but harmless, the first might be adequate. The number that decides is not accuracy; it is the expected cost, which can be written

If c_FN = 100 and c_FP = 1, the first detector costs 100 × 12 + 6 = 1 206 and the second 100 × 2 + 40 = 240, five times lower.

Where evaluations go wrong

Engineering implications and limitations

Write the requirement first. “Detect any person within 8 m in the stopping corridor with recall at least 99.5% at the chosen false-alarm rate, in dry daylight and in rain, latency under 150 ms.” A requirement of this kind produces a test; “detect people well” does not.

Build a scenario library. Collect or simulate the cases that matter: far and near, partially occluded, unusual poses, bad weather, empty scenes. Keep the hard ones as permanent regression tests, and add every field failure to the set.

Measure per condition and per range. An overall number averages over conditions in proportion to how often they happened in your data, not how much they matter. A system that is perfect indoors and poor outdoors can have a good average if most of the data is indoors.

Evaluate the pipeline and the parts. The end-to-end metric tells you whether the robot works; component metrics tell you where to fix it. Use both, and recompute the end-to-end number after every change.

Keep a test set you do not touch. Each time you look at test results and then change the system, the test set becomes a little more like tuning data. A final untouched test set, used rarely, gives an honest estimate.

Include the “no object” case. Empty scenes catch false alarms, which in production are the dominant cost for most robots. A test set built only of positive examples cannot measure precision.

Common misconceptions

Engineering takeaways

1. Start from the task: what the robot does with the output and what each kind of error costs.
2. Precision, recall, accuracy and F1 come from the confusion matrix; accuracy is misleading when positives are rare.
3. Summarise continuous errors by their distribution (median, p95, maximum), not only the mean or RMSE.
4. A measured rate has a standard error √(p(1−p)/n); report counts and intervals.
5. Avoid leakage, unrepresentative data and tuning on the test set; evaluate per condition, and include empty scenes.

References and further reading

The counts in the worked examples are invented for illustration and do not come from a real detector.

• Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A. (2010). The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88: precision, recall and average precision for detection.
• Geiger, A., Lenz, P. and Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. CVPR: an example of a benchmark whose conditions limit what its scores say.
• Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning, 2nd ed. Springer: training, validation and test sets, and the cost of tuning on test data.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes Module 1. Next: the pixels themselves: how an image is stored, how colour is represented and how to work with it.

Related content

Nothing below is required to finish this lesson.

• Before
  P1-T5: Noise, Uncertainty, Occlusion and Ambiguity

  Classify the ways a measurement can be wrong or incomplete and the effect on downstream results.

• Lab
  P1-L3: Build a First Perception Pipeline

  Assemble raw range data into a useful output (is the way clear, and how far?) and evaluate it.