Skip to the lesson
FixTheRobot Explore Robotics / Perception / P6 Object Detection and Semantic Understanding / P6-T2

Perception · Theory · P6

P6-T2 — Bounding Boxes and Confidence Scores

Four numbers, a class and a score per object, and a clean-up step for the duplicates. Each part has its own failure mode.

  • Lesson 2 of 6
  • Module 6: Object Detection and Semantic Understanding
  • Theory
  • About 9 minutes to read

A detector does not say “there is a cone”. It returns a list of boxes, each with a class and a score, and usually many more boxes than objects. Using those boxes needs four things: knowing the box format, understanding what the score does and does not mean, measuring overlap with intersection over union, and removing duplicates with non-maximum suppression. Each has pitfalls that appear as odd robot behaviour. Lab P6-L1 asks you to decode raw arrays, apply a threshold and non-maximum suppression, and relate a class map to boxes; Lab P6-L3 studies the threshold.

Learning objectives

After this lesson, you should be able to:

  • Convert between the common bounding box formats and compute a box’s area.
  • Compute the intersection over union of two boxes and use it to decide overlap.
  • Explain what a confidence score is, why it may not be a probability, and how calibration is checked.
  • Apply non-maximum suppression step by step and describe how its threshold fails on crowded scenes.

The idea: many proposals, one answer per object

A detector looks at many places and sizes at once. Near a real object, several of those looks fire, each with a slightly different box and a slightly different score. The raw output is therefore a cloud of overlapping guesses around each object, plus a few stray guesses where nothing is. Turning this into one box per object takes a rule for how sure to be (the threshold) and a rule for which of the overlapping boxes to keep (non-maximum suppression).

Non-maximum suppressionLeft panel: three overlapping boxes around one object with scores 0.90, 0.80 and 0.70, and one separate box with score 0.60. Right panel: after non-maximum suppression only the 0.90 box and the separate 0.60 box remain, because the other two overlapped the best one with an IoU above the threshold.Before: raw detector outputAfter: IoU threshold 0.5 0.900.800.700.60 0.90 kept0.60 kept The two overlapping lower-score boxes are duplicates of the best one; the separate box is a different object.
Three boxes around one object collapse to the best one; a box on a different object survives because it barely overlaps the winner.

Box formats

The same rectangle can be written in different ways, and mixing them is one of the commonest bugs in detection code. The corner format gives the top-left and bottom-right corners, (x0, y0, x1, y1). The centre format gives the centre and the size, (cx, cy, w, h). Some tools use the top-left corner and the size. Coordinates may be in pixels or normalised to the range 0 to 1, and image rows grow downwards. The labs in this track use pixel corners, [x0, y0, x1, y1], and say so.

What the score means

The score is the model’s confidence in a box. It orders boxes: a higher score means the model is more sure. It is tempting to read 0.8 as “80% of such boxes are right”, but that holds only if the model is calibrated, which most are not by default. Scores also depend on object size, occlusion and lighting: the same object can score 0.9 by day and 0.5 at dusk. A threshold chosen by day can therefore miss objects at dusk, which is what Lab P6-L3 and Lab P6-L4 explore.

Duplicates and the rule that removes them

Because neighbouring places fire together, the same object appears several times. Non-maximum suppression (NMS) keeps the best box and removes the others that overlap it too much. The measure of overlap is intersection over union, which the next section defines.

The mathematics: IoU, decoding and NMS

The overlap between two boxes A and B is the intersection over union:

intersection over union
IoU⁡(A, B)=∣A∩B∣/∣A∪B∣=∣A∩B∣/(∣A∣+∣B∣−∣A∩B∣)\operatorname{IoU}(A,\,B)=\lvert A\cap B\rvert /\lvert A\cup B\rvert =\lvert A\cap B\rvert /(\lvert A\rvert +\lvert B\rvert -\lvert A\cap B\rvert )
|A|
the area of box A, (x1 − x0) × (y1 − y0).
|A ∩ B|
the area of the overlap: width max(0, min(x1ᴬ, x1ᴮ) − max(x0ᴬ, x0ᴮ)) times the same for the height; zero if the boxes are apart.
|A ∪ B|
the area covered by either box, written as the sum of the areas minus the overlap so that the overlap is not counted twice.

IoU is 1 for identical boxes and 0 for disjoint ones. It is symmetric, and it does not depend on the units. It is the standard measure of both duplicate boxes and localisation quality (Lesson P6-T5).

Decoding a raw output

Many detectors do not predict a box directly. For each location they predict four offsets (tx, ty, tw, th) relative to a fixed reference box called an anchor with centre (ax, ay) and size (aw, ah). The box is recovered by

anchor decoding
cx=ax+tx awc_{x}=a_{x}+t_{x}\,a_{w}
cy=ay+ty ahc_{y}=a_{y}+t_{y}\,a_{h}
w=aw exp⁡(tw)w=a_{w}\,\exp (t_{w})
h=ah exp⁡(th)h=a_{h}\,\exp (t_{h})
tx, ty
predicted shift of the centre, in units of the anchor size, so a value of 0.1 moves the centre by a tenth of the anchor width.
tw, th
predicted log scale: 0 keeps the anchor size, positive values enlarge it. The exponential keeps sizes positive.
ax, ay, aw, ah
the anchor: a prior guess at position and size, fixed for each location.

This is the form used by the family of detectors that descend from Faster R-CNN. A bug in the decoding (using the wrong anchor, forgetting the exponential, mixing corner and centre formats) gives boxes that are plausible but in the wrong place, and is invisible to a threshold.

The NMS algorithm

  1. Discard every box with a score below the threshold τ.
  2. Sort the rest by score, highest first.
  3. Take the best box and keep it. Remove every remaining box of the same class whose IoU with it is at least ν.
  4. Repeat from the best of what remains until the list is empty.

Two parameters appear: the score threshold τ, which trades misses for false alarms, and the overlap threshold ν, typically 0.5, which trades duplicates for lost neighbours. A small ν removes duplicates aggressively and also removes true neighbours that overlap; a large ν keeps both.

A worked example: decode, measure, suppress

Decoding. An anchor has centre (50, 40) and size 40 × 30. The network predicts t = (0.1, −0.2, 0.2, 0). The centre moves to cx = 50 + 0.1 × 40 = 54 and cy = 40 − 0.2 × 30 = 34. The width becomes 40 × e0.2 = 48.9 and the height stays 30 × e0 = 30. In corner form the box is (29.6, 19, 78.4, 49).

IoU. Take box A = (10, 10, 50, 50) with score 0.90, box B = (20, 20, 60, 60) with score 0.60, and box C = (12, 12, 52, 52) with score 0.80. Each has area 40 × 40 = 1600.

PairOverlap rectangleIntersectionUnionIoU
A and B(20, 20, 50, 50)30 × 30 = 9003200 − 900 = 23000.391
A and C(12, 12, 50, 50)38 × 38 = 14443200 − 1444 = 17560.822

NMS with ν = 0.5. The best box is A (0.90): keep it. C has IoU 0.822 with A, which is at least 0.5, so C is removed as a duplicate. B has IoU 0.391, which is below 0.5, so B stays. The final list is A and B. Now see what a harsher setting does: with ν = 0.3 B would also be removed (0.391 ≥ 0.3), which is right if B is a duplicate of A and wrong if B is a second object that stands partly behind it. NMS cannot tell the two cases apart from the boxes alone.

Calibration, checked by counting

Suppose 100 detections have scores between 0.75 and 0.85, around 0.80. If the model is calibrated, about 80 of them should be correct. You count and find 62. The model is over-confident by 0.80 − 0.62 = 0.18 in that score range. Repeating this in bins of score gives a reliability table; a threshold based on the raw score then selects fewer correct boxes than expected, which matters when you pick a threshold from a cost (Lab P6-L3).

Engineering implications and limitations

Fix one box format and test it. Write a unit test with a box whose corners you know and check every conversion, including the normalised and the pixel versions, and the direction of the y axis.

Apply NMS per class. A cone and a crate that overlap in the image are different objects; suppressing across classes can delete the correct one. The labs apply it per class.

Tune ν to the scene. In crowds and queues, objects overlap in the image and a 0.5 threshold can remove real ones. Variants such as soft-NMS lower the score of overlapping boxes rather than removing them; the right choice depends on how crowded the data are.

Do not trust scores across conditions. Check the score distribution per condition (Lab P6-L4) and, if the decision depends on it, calibrate on data from the conditions where the robot works.

Keep the raw output. Log the boxes before the threshold and before NMS. When the robot misses something, the log shows whether the detector never saw it, scored it low, or suppression removed it. These are three different faults.

Mind the clock. NMS is cheap on a few hundred boxes, but runs on every frame. In a crowded scene with thousands of candidates, check that it fits the latency budget (Lesson P1-T3).

Common misconceptions

“A score of 0.9 means the box is right 90% of the time.”

Only for a calibrated model. Scores are best treated as a ranking, and the real hit rate at each score has to be measured.

“IoU measures how good a detection is.”

It measures box overlap only. A perfect box with the wrong class is useless, and a box that is a little loose is often fine; the right IoU requirement depends on the task.

“NMS removes false positives.”

It removes duplicates of a box. A confident wrong box on an empty patch survives, and a true object next to a better one may be removed.

“All boxes use the same format.”

Corner, centre, size, pixel and normalised formats are all in use. Mixing them gives wrong but plausible-looking boxes.

Engineering takeaways

  1. A detector returns a variable-length list of boxes with classes and scores; there are usually many boxes per object.
  2. IoU = intersection / (sum of areas − intersection): 1 for identical boxes, 0 for disjoint.
  3. Raw outputs are often offsets relative to anchors and must be decoded: centre shifts scaled by the anchor, sizes by an exponential.
  4. NMS keeps the best box per cluster of same-class overlaps; its IoU threshold trades duplicates against lost neighbours.
  5. Scores rank boxes but are not probabilities unless calibrated; check calibration by counting hits per score range.

References and further reading

Papers cited for the methods named. The boxes and the 62-in-100 calibration count are constructed for the example.

  • Neubeck, A. and Van Gool, L. (2006). Efficient Non-Maximum Suppression. ICPR.
  • Ren, S., He, K., Girshick, R. and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NeurIPS: anchors and offset decoding.
  • Guo, C., Pleiss, G., Sun, Y. and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML.
  • Bodla, N., Singh, B., Chellappa, R. and Davis, L. S. (2017). Soft-NMS: Improving Object Detection With One Line of Code. ICCV.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

Box P = (0, 0, 20, 20) and box Q = (10, 0, 30, 20). What is their IoU?

Explanation

Each has area 400. The overlap is (10, 0, 20, 20) = 10 × 20 = 200. The union is 800 − 200 = 600, so the IoU is 200 / 600 = 0.333.

Question 2

A detector gives three boxes of the same class with scores 0.9, 0.85 and 0.4. The IoU of the 0.9 box with the 0.85 box is 0.7 and with the 0.4 box is 0.1. With ν = 0.5 and τ = 0.5, which boxes remain?

Explanation

The 0.4 box is removed first by the score threshold τ. The 0.85 box overlaps the best box with IoU 0.7 ≥ 0.5, so it is suppressed. Only the 0.9 box remains. (If the 0.4 box has IoU 0.1 it would have survived NMS but had already failed the score threshold.)

Question 3

In a crowd, NMS removes a real pedestrian standing behind another. Name two changes that might help and one risk of each.

Explanation

Raise ν (say to 0.6) so that only near-identical boxes are removed; the risk is that more true duplicates survive. Use soft-NMS, which lowers the score of overlapping boxes instead of deleting them; the risk is that a duplicate with a reduced score may still pass the score threshold.

What’s next

Next: when boxes are not enough, pixel-level outputs: semantic and instance segmentation.

Practice this lesson

Interpret Detection Outputs

Match predicted boxes to ground truth with IoU and non-maximum suppression.

Next lesson

P6-T3 — Semantic and Instance Segmentation

Compare per-pixel class maps with per-object masks and when each is needed.

Nothing below is required to finish this lesson.