P6-T5 — Precision, Recall and IoU

Learning objectives

After this lesson, you should be able to:

• Define true positive, false positive and false negative for detections using an IoU threshold, class and one-to-one matching.
• Compute precision, recall and F1, and explain why they depend on the score threshold.
• Build a precision-recall curve from scored detections and compute average precision with the envelope.
• Explain how the IoU threshold and averaging rules change reported numbers and why they must be stated.

The idea: count carefully, then trace the trade-off

Everything starts with a matching. The detector has a list of predictions; the dataset has a list of true objects. You pair them up, and each pair is a success. The rest are failures of two kinds: predictions with no partner, and objects with no partner.

The matching rule

1. Sort the predictions by score, highest first.
2. For each prediction, look at the true objects of the same class that are not yet matched.
3. If the best IoU among them is at least the IoU threshold ν, match the prediction to that object: a true positive (TP), and the object is used up.
4. Otherwise the prediction is a false positive (FP): a wrong class, a poorly placed box, a duplicate of an object already matched, or a box on nothing.
5. Every object left unmatched at the end is a false negative (FN), a miss.

Two details decide many results. A duplicate counts as an error even though it covers a real object, which is why non-maximum suppression (Lesson P6-T2) matters for the score. And the matching is one to one, so counts always satisfy TP + FN = the number of true objects.

Why the IoU threshold is part of the metric

A box that overlaps its object with IoU 0.6 is a true positive at ν = 0.5 and a false positive at ν = 0.75. The same detector, run once, therefore has different precision and recall at different thresholds. A loose threshold answers “is it roughly there?”, a strict one “is it located accurately?”. A grasping robot needs the second; a counting task needs the first. Stage 1 of the lab lets you see the numbers move with nothing but the rule changed.

From a point to a curve

All the counts above are for one score threshold. Lowering that threshold admits more predictions: recall can only go up, while precision usually goes down, since the new predictions are less certain. Plotting precision against recall as the threshold sweeps from high to low gives the precision-recall curve. It shows what the detector can do at every operating point, and the area under it summarises the whole trade-off in one number.

The mathematics: precision, recall, F1 and average precision

With the counts TP, FP and FN at one score threshold and one IoU threshold:

TP + FP

the number of predictions kept above the score threshold.

TP + FN

the number of true objects; this does not change with the score threshold.

precision

the fraction of the predictions that are right: how much to trust a detection.

recall

the fraction of the objects that were found: how much of the world is covered.

F1

the harmonic mean of the two; it is low if either is low, and it assumes both errors matter equally, which is often false in robotics.

Now sort all predictions of one class by score and walk down the list. After each prediction k, compute the recall r_k and precision p_k using the first k predictions. The curve is the set of points (r_k, p_k). Because precision zig-zags, we replace it by the envelope, the best precision available at that recall or at any higher recall:

r_k, p_k

recall and precision after the k highest-scoring predictions; r₀ = 0.

p_env

the non-increasing envelope of the precision, which removes the zig-zag and makes the area less sensitive to the order of a few predictions.

AP

the area under the envelope, between 0 and 1, computed as the sum of strips of width r_k − r_k−1.

Mean average precision (mAP) averages AP over the classes. Benchmarks add a further average: the widely used COCO protocol averages AP over ten IoU thresholds, 0.50, 0.55, …, 0.95, which rewards accurate localisation, while the older PASCAL VOC protocol uses 0.5 only. Interpolation details also differ between benchmarks, so numbers from two benchmarks are not comparable.

A worked example: five detections, three objects

One class has 3 true objects. The detector gives five predictions. After matching at one IoU threshold, in order of decreasing score, they are:

The envelope. Going from the end: the last precision is 0.600. At k = 4, the best precision at this recall or higher is max(0.500, 0.600) = 0.600. At k = 3 it is max(0.667, 0.600) = 0.667; at k = 2 it is max(0.500, 0.667) = 0.667; at k = 1 it is max(1.000, 0.667) = 1.000.

The area. The recall steps are 1/3 at k = 1, 0 at k = 2, 1/3 at k = 3, 0 at k = 4 and 1/3 at k = 5. Therefore

Without the envelope, using the raw precisions, the area would be 1/3 × 1.000 + 1/3 × 0.667 + 1/3 × 0.600 = 0.756 here as well, because the zig-zag points sit at recall steps of zero. They differ when a false positive comes before a recall step, as in the order (FP, TP, TP, TP): the raw precisions 0, 0.5, 0.667, 0.75 would give 1/3 × (0.5 + 0.667 + 0.75) = 0.639, and the envelope (0.75 everywhere) gives 0.75 × 1 = 0.75 on the three steps, an area of 0.75. The two rules answer slightly different questions, which is why a metric must be quoted with its definition.

One operating point. Keeping only the predictions with score at least 0.75 keeps k = 1, 2, 3: TP = 2, FP = 1, FN = 1. Precision 0.667, recall 0.667, F1 = 0.667. Keeping score at least 0.55 keeps all five: TP = 3, FP = 2, FN = 0, precision 0.6, recall 1.0, F1 = 2 × 0.6 × 1.0 / 1.6 = 0.75.

Engineering implications and limitations

Write the definition next to every number. “mAP 0.62” means little. “AP at IoU 0.5, all-point envelope, classes cone/crate/barrel, test set of 8 drives” can be reproduced.

Choose the IoU to match the task. Counting or presence: 0.5 may do. Picking, docking, following a path: ask for 0.75 or higher and check that the labels are consistent enough to support it (Lesson P6-T4).

Do not stop at AP. AP summarises all thresholds; the robot runs at one. Report precision, recall and the number of false alarms per hour or per kilometre at the operating point, which is what the safety case needs (Lesson P6-T6).

Test the evaluation code. Matching rules are easy to get subtly wrong: forgetting that an object can be matched once, ignoring the class, or breaking ties differently. Check them on tiny cases computed by hand, like the example above. Lab P6-L2 does exactly this.

Report uncertainty. Recall measured on 40 objects carries an interval of about ±0.09 (Lesson P6-T4). A difference of 0.02 between two detectors on a small test set is not a result.

Keep the honest label. In the lab the predictions come from a simulated detector, so the AP values show how the metric behaves, not how any real network scores.

Common misconceptions

Engineering takeaways

1. Match predictions to objects one to one, by score order, class and IoU at least ν; unmatched predictions are FP and unmatched objects are FN.
2. Precision = TP/(TP+FP), recall = TP/(TP+FN); both depend on the score and IoU thresholds.
3. The precision-recall curve is traced by lowering the score threshold; AP is the area under its envelope, mAP the mean over classes.
4. In the example the five detections give AP = 0.756; F1 at the cut-off 0.75 is 0.667 and at 0.55 is 0.75.
5. Always state the IoU threshold, the averaging rule and the data with every reported number.

References and further reading

Papers and benchmark protocols named in the text. The five-detection table is constructed for the worked example.

• Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A. (2010). The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision: precision, recall and AP for detection.
• Lin, T.-Y. et al. (2014). Microsoft COCO: Common Objects in Context. ECCV: AP averaged over IoU thresholds 0.50 to 0.95.
• Davis, J. and Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. ICML.
• Hoiem, D., Chodpathumwan, Y. and Dai, Q. (2012). Diagnosing Error in Object Detectors. ECCV: breaking detector errors into types.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what the errors mean in practice, false positives, false negatives, and what happens when the scene differs from the data.

Related content

Nothing below is required to finish this lesson.

• Before
  P6-T4: Training Data and Annotation

  Describe how labelled data define what a model can learn and the ways annotations go wrong.

• Lab
  P6-L2: Evaluate Predictions Against Ground Truth

  Compute precision, recall and average precision from a set of predictions.

• Lab
  P6-L3: Investigate Confidence Thresholds and Missed Detections

  Choose a threshold for a safety task and see the effect on misses and false alarms.