Validating detection accuracy

1. Engineering problem

“The detector works” is not a result. A number means something only with a ground-truth set, a matching rule and a threshold. Precision and recall show different failures: false alarms and misses.

2. When is this needed?

• Before deploying a detector or a new calibration.
• Comparing two pipelines.
• After changing thresholds.

3. Selection and design decisions

Standard definitions: a prediction is a true positive (TP) if it overlaps an unmatched ground-truth object with IoU at or above the threshold; unmatched predictions are false positives (FP); unmatched ground-truth objects are false negatives (FN). Precision = TP ÷ (TP + FP); recall = TP ÷ (TP + FN). Each ground-truth object may be matched only once, so duplicate predictions count as false positives.

4. Advantages and disadvantages

• High threshold: better box accuracy, lower reported scores.
• Safety-critical classes favour recall; nuisance-sensitive ones favour precision.
• Larger labelled sets give tighter estimates and cost labelling time.

5. Requirements and constraints

• Ground truth is independent of the system under test.
• The dataset covers the operating conditions (range, light, weather).
• The matching rule and IoU threshold are published with the result.

6. Architecture

predictions + ground truth ─▶ match by IoU (one-to-one) ─▶ TP, FP, FN ─▶ precision, recall (per range and class)

7. Implementation

1. Label ground truth for representative recordings.
2. Run the pipeline on the recordings and collect predictions.
3. Match one-to-one by IoU (below).
4. Compute precision and recall overall and per range and class.
5. Record the dataset, threshold and software version with the result.

def iou(a, b):
    x0, y0 = max(a[0], b[0]), max(a[1], b[1]); x1, y1 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, x1 - x0) * max(0, y1 - y0)
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)
truth = [(0, 0, 10, 10), (20, 20, 30, 30), (50, 50, 60, 60)]
preds = [(1, 1, 10, 10), (21, 19, 31, 29), (80, 80, 90, 90), (0, 0, 9, 11)]       # last is a duplicate of truth 0
tp, used = 0, set()
for p in preds:
    best = max(((iou(p, t), j) for j, t in enumerate(truth) if j not in used), default=(0, None))
    if best[0] >= 0.5:
        tp += 1; used.add(best[1])
fp, fn = len(preds) - tp, len(truth) - tp
print(f"TP {tp}, FP {fp}, FN {fn}, precision {tp / (tp + fp):.2f}, recall {tp / (tp + fn):.2f}")

Printed (checked by the site’s tests):

TP 2, FP 2, FN 1, precision 0.50, recall 0.67

Reading it: of four predictions two match ground truth (TP), two do not (a far-away box and a duplicate of an already matched object), and one ground-truth object is missed, giving precision 0.50 and recall 0.67. The boxes are illustrative.

8. Real examples

• Pedestrian detector: precision and recall per distance band show recall falling beyond 15 m, which sets the safe speed.

9. Failure scenarios

• Testing on the training data.
• A dataset taken only in daylight.
• Reporting a score without the threshold.

10. Troubleshooting

1. Look at the FPs and FNs visually.
2. Check duplicates and matching rule.
3. Break down by range and condition.

11. Validation

• The example prints the line shown (executed by the site tests).
• The result is stated with dataset, threshold and version, and is reproducible.

12. Engineering checklist

• Independent ground truth
• Representative data
• One-to-one matching
• Threshold stated
• Metrics per range and class
• Result reproducible

13. Further references

• Precision, recall and IoU are standard definitions; benchmarks may use further variants such as average precision, whose exact definition depends on the benchmark.

Related on FixTheRobot