P6-T1 — Classification, Detection and Segmentation

Learning objectives

After this lesson, you should be able to:

• Distinguish image classification, object detection and semantic or instance segmentation by their outputs.
• Compute class probabilities from scores with softmax and the loss of a prediction.
• Compare the size of the output of each task and explain why it matters for labelling and compute.
• Choose the least expensive task that supports a given robot decision.

The idea: ask for as little as the decision needs

Imagine asking three questions about the same picture. “Is there a traffic cone in view?” needs one word. “Where are the cones, and how many?” needs a position for each. “Which pixels can the robot drive on?” needs a decision for every part of the image. These are the three tasks of this lesson.

Image classification

The model reads the whole image and returns one class, usually with a probability for every possible class. It does not say where anything is, and it cannot count: an image of five cones and an image of one cone both give “cone”. It is the cheapest task to label, because a person only has to name what the picture shows. It suits questions about the scene as a whole: is the floor wet, is the gripper holding something, is it day or night.

Object detection

The model returns a list. Each entry is a bounding box (four numbers), a class and a confidence score. The list has a variable length: zero entries when nothing is found, ten in a busy scene. Detection answers “what and where”, so it supports counting, tracking, distance estimation with known object sizes, and picking one object to approach. Labelling is more expensive, since a person draws a box around every object. Lessons P6-T2 and P6-T5 are about boxes, scores and how to evaluate them.

Semantic and instance segmentation

Semantic segmentation assigns a class to every pixel. The result is a map, the same size as the image, and it describes regions of any shape: drivable road, sky, sidewalk. It does not separate two touching objects of the same class; the pixels of two adjacent crates are simply “crate”. Instance segmentation gives each object its own mask, and so separates the crates, but covers only things that can be counted, not amorphous regions such as road. Lesson P6-T3 compares them. Labelling is the most expensive, because a person outlines the regions.

These tasks are nested, not rival. A detector’s box contains information a classifier lacks, and a segmentation mask gives a tighter outline than a box. A pipeline often uses several: a segmentation map to find free space and a detector to find the people in it.

The mathematics: scores, probabilities and output sizes

A classifier does not output a probability directly. The network’s last layer produces one raw score, a logit, for each of the K classes, and the softmax function turns the scores into numbers that are positive and sum to one:

z_k

the raw score (logit) the network gives to class k; it can be any real number.

K

the number of classes; j runs over all of them in the sum.

p_k

the resulting probability of class k, between 0 and 1; the K values add up to 1.

Training compares the probability given to the true class with the truth using the cross-entropy loss of one example:

p_true

the probability the model gave to the class that is actually correct.

L

the loss: 0 when p_true = 1 and growing without bound as p_true approaches 0.

A model that is confident and right pays almost nothing; one that is confident and wrong pays heavily. This is why the output of a network behaves like a confidence: it was trained to put probability on the right answer. Whether that confidence matches how often the model is actually right is a separate question, discussed in Lessons P6-T2 and P6-T6.

How big is each output?

For an image of H × W pixels and K classes:

The counts explain the costs. A class label is the quickest annotation to produce, a box takes longer and an outline longer still; the segmentation output carries about 100,000 times more numbers than a classification (307,200 against 3 in the table), and has to be computed, stored and transmitted at the camera rate. The decision about the task is partly a decision about budget.

A worked example: from scores to a decision

A classifier looks at an image and gives raw scores z = (2.0, 1.0, 0.1) for the classes cone, crate and barrel. First the exponentials: e^2.0 = 7.389, e^1.0 = 2.718 and e^0.1 = 1.105. Their sum is 11.212, so

The top class is “cone” with probability 0.66. The loss, if the true class is cone, is −ln 0.659 = 0.417; if the truth were barrel the loss would be −ln 0.099 = 2.31, more than five times larger. Note what the model is not saying: with a probability of 0.66 it is wrong about one time in three when it is calibrated, and the three numbers add to one by construction, so even an image of a chair would receive three probabilities that sum to one. The classifier has no way of answering “none of these” unless a class for it is added.

Choosing the task for a robot decision

A warehouse robot must (a) stop if the floor is blocked, (b) count pallets on a shelf, and (c) find the free floor to drive on. The table maps each to the cheapest sufficient output.

Two lessons follow. First, ask the question before choosing the model. Second, the cheapest sufficient output is also the easiest to evaluate, since fewer things can go wrong in it.

Engineering implications and limitations

Start from the decision. Write the sentence “the robot will do X if the perception says Y”. Y is the output you need, in the units and at the rate the decision uses.

Do not use a classifier as a detector by accident. Cropping windows and classifying each one works, but turns a cheap task into an expensive one with duplicate hits. If you need positions, use a model built to give them.

Add a “none of the above” class. A closed set of classes forces an answer. Include a background or unknown class, and test the model on inputs from outside its classes to see what it says.

Budget the output size. A segmentation map at camera resolution on a small processor can dominate the compute budget. Run it at a lower resolution, or only inside a region of interest, if the decision allows.

Keep the interface simple. A downstream module that expects a list of boxes should not receive a map. Fix the data type, the frame, the units and the meaning of “nothing found” as in Lesson P1-T3.

Remember what you are studying here. The labs in this module use a detector simulator that produces detector-like output from known truth with an explicit error model. That is the right tool to study evaluation, but it is not a trained network, and its numbers say nothing about a real model’s accuracy.

Common misconceptions

Engineering takeaways

1. Classification gives one label, detection a variable list of boxes with classes and scores, segmentation a class for every pixel.
2. Softmax turns raw scores into probabilities that sum to one; cross-entropy is −ln of the probability given to the truth.
3. The output sizes differ by orders of magnitude, and so do the labelling and compute costs.
4. Pick the cheapest task that supports the robot’s decision; start from the decision, not the model.
5. The labs in this module use a simulated detector with a stated error model, not a trained network.

References and further reading

Textbook and benchmark references. The scores and the warehouse questions in the example are illustrative.

• Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning. MIT Press: softmax and cross-entropy.
• Lin, T.-Y. et al. (2014). Microsoft COCO: Common Objects in Context. ECCV: a standard benchmark with boxes and masks.
• Long, J., Shelhamer, E. and Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: recognition tasks and their outputs.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what a detector actually outputs, bounding boxes and confidence scores, and how non-maximum suppression removes duplicates.

Related content

Nothing below is required to finish this lesson.

• Before
  P5-T6: Occlusion, Sparse Measurements and Density Changes

  Predict how range, occlusion and sparsity change what a detector can see.