P6-T3 — Semantic and Instance Segmentation

Learning objectives

After this lesson, you should be able to:

• Distinguish semantic, instance and panoptic segmentation by what each output labels.
• Explain why a semantic map merges adjacent objects of the same class and how instance masks or connected components separate them.
• Compute pixel accuracy, per-class IoU and mean IoU from a confusion matrix.
• Choose between a mask and a box for a given robot task.

The idea: outlines instead of rectangles

A bounding box is a rectangle that is a good fit for compact objects and a poor fit for everything else: a long cable, a diagonal pallet, a curved road edge. A mask marks exactly the pixels that belong to the object. The price is the amount of data (one decision per pixel) and a harder labelling job.

Semantic segmentation

Every pixel gets a class from a fixed list: road, sidewalk, person, crate, background. The output is a class map the size of the image. It gives free space, boundaries and the extent of regions. It does not count objects. If two crates touch, their pixels carry the same label and form one region, as in the left grid of the figure.

Instance segmentation

Each thing (a countable object) gets its own mask and its own identity: crate 1 and crate 2. Instance methods typically predict a box and a mask inside it for each object, a design popularised by Mask R-CNN. They do not label “stuff” (road, sky, grass), which has no natural count.

Panoptic segmentation

Panoptic segmentation combines the two: every pixel gets a class, and pixels of countable classes also get an instance identity. It is the most complete description, and the most expensive to label.

From a class map to objects

You can often recover instances from a semantic map with the tools of Module 2. Take the pixels of one class and find their connected components: each component is a candidate object, with a box, an area and a centre. This works when objects of the class are separated by other pixels and fails when they touch, which is where the merge in the figure comes from. You can then check the result with the knowledge you have, for example that a crate has an area between two limits, and split or flag a region that is too large.

The mathematics: pixel accuracy and mean IoU

Evaluation starts from a confusion matrix M in which entry M_ij counts the pixels whose true class is i and whose predicted class is j. The diagonal holds the correct pixels. Pixel accuracy is the fraction of all pixels on the diagonal:

It is simple and easy to inflate: if most of the image is road, a model that labels everything as road scores high. A better measure treats each class separately. The IoU of class c is the overlap of its true and predicted regions divided by their union (the same quantity as for boxes in Lesson P6-T2, now on pixel sets):

M_cc

pixels of class c correctly labelled c (true positives).

Σ_j M_cj

all pixels whose true class is c (row sum): true positives plus misses.

Σ_i M_ic

all pixels predicted as c (column sum): true positives plus false alarms.

IoU_c

between 0 and 1; the denominator counts each pixel of the union once, hence the subtraction of M_cc.

The mean IoU (mIoU) averages IoU_c over the classes, so every class counts equally, however few pixels it has:

For instance segmentation the metrics are those of detection (Lesson P6-T5) with mask IoU in place of box IoU, because the objects are matched one by one.

A worked example: 80% accuracy and a poor cone class

A 10 × 10 image (100 pixels) has three classes: road, cone and background. The confusion matrix has rows for the true class and columns for the prediction:

Pixel accuracy. The diagonal is 54 + 6 + 20 = 80, so the accuracy is 80 / 100 = 0.80.

Per-class IoU.

Mean IoU. (0.750 + 0.500 + 0.556) / 3 = 0.602.

The accuracy of 0.80 looks acceptable, but the mean IoU of 0.60 is lower and the cone class is only half right: of the ten cone pixels, four were labelled road, and the model painted two background pixels as cone. For a robot that must avoid cones, the road class is the large, easy part and the cone class is the one that matters. Accuracy is dominated by the large class; mIoU and the per-class table expose the weak one.

From mask to box

A mask gives a box for free: the smallest rectangle that contains the mask pixels. The reverse does not work, since a box does not say which pixels inside belong to the object. If the cone mask above has its pixels in columns 3 to 5 and rows 4 to 8, the box is (3, 4, 6, 9) in half-open pixel coordinates, an area of 3 × 5 = 15 pixels while the mask has only 10. A box is a looser description, and it includes five pixels that are not cone.

Engineering implications and limitations

Use segmentation where shape matters. Free space, lane markings, crack detection, grasp regions and anything long or thin gain from masks. Compact objects counted for tracking are usually served by boxes.

Check classes separately. Report the IoU of each class and look at the confusion matrix. A rare class with a poor IoU can hide behind a good average, and the confusion pattern says what it is mistaken for.

Be careful at boundaries. Pixels at the edge of an object are ambiguous even for a human labeller, and a thin object has a large boundary share. IoU on a thin cable is much lower than on a large crate for the same visual quality of the outline.

Post-process masks. Remove tiny components, fill holes and keep components that have a plausible area. Each rule is a threshold with its own failure mode, as in the classical pipelines of Module 2, and each should be evaluated.

Mind the resolution. Masks are often predicted at a lower resolution than the image and enlarged. The error in the outline is then a few pixels, which matters for a small object at a distance.

Keep the honest label. Lab P6-L1 gives you a class map made by a simulator from known truth, with stated errors. It teaches reading the output, not what a trained segmentation network does.

Common misconceptions

Engineering takeaways

1. Semantic segmentation labels each pixel with a class; instance segmentation gives each object its own mask; panoptic does both.
2. Touching objects of the same class merge in a semantic map; connected components or instance masks separate them, subject to checks.
3. Pixel accuracy = diagonal / total; IoU_c = M_cc / (row + column − M_cc); mIoU averages over classes.
4. Accuracy can be high while an important small class is poor; report the per-class IoU and the confusion matrix.
5. A mask gives a box (its bounding rectangle) but a box does not give a mask.

References and further reading

Papers for the methods and metrics named. The confusion matrix in the example is constructed for teaching and is not from a real model.

• Long, J., Shelhamer, E. and Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR.
• He, K., Gkioxari, G., Dollár, P. and Girshick, R. (2017). Mask R-CNN. ICCV.
• Kirillov, A., He, K., Girshick, R., Rother, C. and Dollár, P. (2019). Panoptic Segmentation. CVPR.
• Everingham, M. et al. (2010). The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision: the IoU measure for segmentation.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: where the labels come from, training data and annotation, and how data quality limits what any evaluation can say.

Related content

Nothing below is required to finish this lesson.

• Before
  P6-T2: Bounding Boxes and Confidence Scores

  Interpret boxes, scores and non-maximum suppression.

• Lab
  P6-L1: Interpret Detection Outputs

  Match predicted boxes to ground truth with IoU and non-maximum suppression.