P6-T4 — Training Data and Annotation

Learning objectives

After this lesson, you should be able to:

• Describe how a labelled perception dataset is produced and where its errors come from.
• Estimate how much label noise lowers a measured accuracy and how annotators’ disagreement limits a strict IoU requirement.
• Compute the uncertainty of a measured recall for a given number of test objects.
• Split data by sequence or scene rather than by frame, and explain why.

The idea: data are a measurement too

Every labelled example is a person’s (or a program’s) judgement about an image: “this is a cone”, “this box is the cone”. The judgement is made quickly, under guidelines, by people who disagree at the edges. A dataset is a large collection of such judgements, and training a model on it teaches the model to reproduce them, mistakes included. Testing the model on it measures agreement with them.

Where labels come from

1. Collect. Choose where and when to record: the sensors, the places, the lighting, the weather, the robot’s own motions. Everything not recorded is invisible to the model.
2. Select. Pick frames to label. Consecutive frames are nearly identical, so picking every frame wastes effort and picking at random misses rare events.
3. Define. Write guidelines: what counts as a cone, how to box an object that is partly hidden, whether to label objects smaller than 10 pixels, what to do with reflections.
4. Label. Annotators draw boxes or masks, often with model assistance, followed by review.
5. Audit. Measure agreement between annotators, review a sample of labels, and fix systematic errors.

Each step shapes the result. Guidelines decide the answer in ambiguous cases; collection decides what the model can know; selection decides which cases are common in training.

Label noise and disagreement

Annotators make three kinds of mistakes: a wrong class, a box that is too loose or too tight, and a missing object. A class error looks like a model error when you score against it, and a model that is right and the label that is wrong count as a miss. There is also honest disagreement: two careful people box the same cone slightly differently, and a strict IoU requirement asks for more accuracy than the labels contain.

Imbalance

Real scenes have many easy, common things and a few rare, important ones. A dataset with 98% of its objects in two classes gives the third very little to learn from and very little to be tested on. A model can look excellent on average and fail on the rare class.

Splitting without leakage

The data are split into training data, used to fit the model, validation data, used to choose settings such as the threshold, and test data, used once at the end. The split must separate things that are alike. Frames from one drive, recorded a tenth of a second apart, are almost the same picture; if some are in training and some in the test set, the test measures memory, not generalisation.

The mathematics: what noise, size and agreement do to numbers

Label noise. Suppose a model is right with probability a on the true class, and the test labels are wrong with probability ε, a wrong label being one of the other K − 1 classes chosen uniformly. If the model’s errors are also spread uniformly over the other classes and independent of the label errors, the measured accuracy is

a

the true accuracy of the model against perfect labels.

ε

the probability that a test label is wrong.

K

the number of classes. A wrong model prediction and a wrong label agree with probability 1 / (K − 1) under the uniform assumption.

The first term is the model being right on a correct label; the second is the small chance that a wrong model and a wrong label happen to coincide. The measured accuracy is lower than the true one, and a better model suffers a bigger absolute loss.

Sample size. A measured recall is a proportion p̂ estimated from n test objects. Its standard error and an approximate 95% interval are

p̂

the measured fraction of objects found.

n

the number of true objects in the test set (for that class or condition), not the number of images.

1.96

the multiplier of the standard error for a 95% normal-approximation interval; the approximation is poor for very small n or for p̂ close to 0 or 1.

Solving for the interval half-width h gives the number of objects needed, n = 1.96² p̂(1 − p̂) / h².

A worked example: how much can you believe 0.80?

Noise. A three-class model has a true accuracy of 0.90. Its test labels are wrong 10% of the time. Then

The measured accuracy is 0.815, 8.5 points below the truth, from label errors alone. A weaker model with a true accuracy of 0.88 would measure 0.88 × 0.90 + 0.12 × 0.05 = 0.798: the true gap of 0.02 between the two models shrinks to 0.017 in the measurement, and neither measured number can be read as the model’s real accuracy.

Sample size. A model finds 80% of the objects of a class. With 50 test objects, SE = √(0.8 × 0.2 / 50) = 0.0566 and the 95% interval is 0.80 ± 0.111, from 0.69 to 0.91. With 500 objects, SE = 0.0179 and the interval is 0.80 ± 0.035. To pin recall to ± 0.05 you need about 1.96² × 0.16 / 0.05² ≈ 246 objects of that class and condition. A rare class with 100 objects (2% of 5000) gives ± 0.078 at best; if you also split by four conditions, each cell has far fewer.

Annotator agreement. Two annotators box the same cone as (10, 10, 50, 50) and (12, 11, 53, 52). The first has area 1600, the second 41 × 41 = 1681, the overlap is 38 × 39 = 1482, so the union is 1600 + 1681 − 1482 = 1799 and the IoU is 1482 / 1799 = 0.82. Two careful humans agree at IoU 0.82 here. An evaluation that requires 0.9 would count one human’s box as an error against the other’s, and no model can be reliably scored there.

Splitting. A dataset has 20 drives of about 500 frames each, 10,000 frames in all. A split by drive might use 14 drives for training (7000 frames), 3 for validation (1500) and 3 for testing (1500). Splitting the frames at random in the same proportions would put frames from every drive in every set, and a frame in the test set would have a near twin in training.

Engineering implications and limitations

Write the guidelines first and keep them. Ambiguous cases decide the labels. Include pictures of edge cases, and record the version of the guidelines with the dataset.

Measure agreement. Have a sample labelled twice and report the IoU and class agreement. It sets the ceiling for any evaluation.

Audit the test set hardest. A few wrong labels in the training set do little harm; wrong labels in the test set change every number you report. Look at the model’s most confident “errors”: they are often label errors.

Split by what must generalise. By drive, site or day if the robot will go to new places; by sensor unit if hardware varies. Never put near-duplicates in both training and test sets.

Tag the data. Record the condition (lighting, weather, location, sensor) with each frame so that you can evaluate by condition (Lesson P6-T6, Lab P6-L4). Untagged data cannot be sliced.

Collect the rare and the dangerous on purpose. Random driving yields few examples of rare events. Plan for them, and report how many examples exist.

Be honest about the lab data. In the labs, ground truth comes from the simulator, so there is no label noise, and the detector is a stated error model. Real systems have both noisy labels and a real model, and the numbers here are only illustrative.

Common misconceptions

Engineering takeaways

1. A dataset is a set of judgements made under guidelines; collection, selection, definition and review each shape what the model learns and what the test measures.
2. Label noise lowers measured accuracy: a_measured = a(1 − ε) + (1 − a)ε/(K − 1); in the example, 0.90 measured as 0.815.
3. Annotators disagree: two careful boxes can have IoU 0.82, which caps how strict an IoU requirement can be.
4. A measured recall has SE = sqrt(p̂(1 − p̂)/n); about 246 objects are needed for ±0.05 at p̂ = 0.8, per class and per condition.
5. Split by sequence, place or day, not by frame, and tag each frame with its condition so results can be sliced.

References and further reading

Papers on dataset construction, bias and label errors. The numerical examples are constructed for teaching and are not measurements of any dataset.

• Lin, T.-Y. et al. (2014). Microsoft COCO: Common Objects in Context. ECCV: annotation protocol and statistics of a large detection dataset.
• Russakovsky, O. et al. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision: large-scale annotation and its issues.
• Torralba, A. and Efros, A. A. (2011). Unbiased Look at Dataset Bias. CVPR.
• Northcutt, C. G., Athalye, A. and Mueller, J. (2021). Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS Datasets and Benchmarks.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the metrics that summarise a detector’s results: precision, recall, IoU and average precision.

Related content

Nothing below is required to finish this lesson.

• Before
  P6-T3: Semantic and Instance Segmentation

  Compare per-pixel class maps with per-object masks and when each is needed.

• Lab
  P6-L4: Compare Performance Across Scene Conditions

  Measure how a detector degrades across lighting, distance and clutter and report it honestly.