Skip to the lesson
FixTheRobot Explore Robotics / Perception / P2 Image Processing for Robotics / P2-T4

Perception · Theory · P2

P2-T4 — Thresholding and Segmentation

Segmentation splits an image into regions. Thresholding is the simplest way, and its failures teach the most.

  • Lesson 4 of 6
  • Module 2: Image Processing for Robotics
  • Theory
  • About 10 minutes to read

To find an object a robot first has to decide which pixels belong to it. Segmentation groups pixels into regions that share a property. The simplest rule is a threshold: pixels brighter than a value are foreground. This lesson shows how to choose a threshold from the image itself with Otsu’s method, when a single threshold fails, how connected components turn a mask into separate objects, and how a little morphology removes the debris.

Learning objectives

After this lesson, you should be able to:

  • Choose a global threshold from a histogram and justify it, and compute Otsu’s threshold for a small histogram.
  • Explain why a global threshold fails under uneven light, and use a local (adaptive) threshold instead.
  • Label connected components in a mask and filter them by area.
  • Apply erosion and dilation (opening and closing) to clean a mask and state their side effects.

The idea: let the histogram show where the classes are

If a scene contains dark floor and bright objects, the pixel values form two clusters, and the histogram of the image shows two humps. A threshold between the humps separates the classes. That is the entire idea of global thresholding: look at the distribution of values and cut where the classes are least mixed.

A bimodal histogram with a threshold between the modesA histogram of 16 grey-level bins with two humps: a tall one on the dark side (the floor) and a smaller one on the bright side (the objects). A vertical dashed line between the humps marks the threshold chosen by Otsu method.048121608162432grey level (16 bins)pixels (count)threshold (Otsu)floorobjects
A histogram with two modes: a dark floor and brighter objects. The threshold sits in the valley between them.

Three steps turn the result into something a robot can use:

  1. Threshold to obtain a binary mask (foreground or background per pixel).
  2. Clean the mask with morphological operations (below) to remove specks and fill holes.
  3. Label connected components to get one region per object, with its area, bounding box and centre.

A threshold decides each pixel on its own. Components decide which pixels belong together. Objects need both.

When one number is not enough

The histogram trick works when the two classes occupy separate ranges. It fails if the light varies across the image (the floor in the shaded corner is darker than an object in the bright corner), if noise is large (the humps merge) or if the object and floor have the same brightness but different colours (use colour, Lesson P2-T2). Lab P2-L2 showed the first case: a fixed threshold works in one light and fails in another, and a threshold relative to a local mean follows the light.

The mathematics: Otsu, adaptive thresholds, components and morphology

Otsu’s method

Otsu’s method chooses the threshold t that best separates the histogram into two classes by maximising the variance between the classes. Let pi be the fraction of pixels at level i. For a candidate t:

Otsu between-class variance
w0(t)=∑i≤t piw0(t)=\sum _{i\le t}\,p_{i}
w1(t)=1−w0(t)w1(t)=1-w0(t)
μ0(t)=∑i≤t i piw0(t)\mu 0(t)=\frac{\sum _{i\le t}\,i\,p_{i}}{w0(t)}
μ1(t)=∑i>t i piw1(t)\mu 1(t)=\frac{\sum _{i>t}\,i\,p_{i}}{w1(t)}
σB2(t)=w0(t) w1(t)(μ0(t)−μ1(t))2\sigma _{B}^{2}(t)=w0(t)\,w1(t)(\mu 0(t)-\mu 1(t))^{2}
t∗=arg max⁡t σB2(t)t^{*}=\operatorname*{arg\,max}_{t}\,\sigma _{B}^{2}(t)
pi
fraction of pixels with grey level i (the normalised histogram).
w0, w1
the weights of the two classes: the fraction of pixels at or below the threshold and above it.
μ0, μ1
the mean grey level of each class.
σB²
the between-class variance: large when the two classes are far apart and both are substantial. It is maximal when the classes are best separated.

The method assumes a histogram with two modes and works poorly if one class is tiny (a few objects in a large floor): the large class then dominates the variance and the threshold drifts towards it.

Adaptive (local) thresholds

A local threshold compares each pixel with a statistic of its own neighbourhood, typically the mean or a Gaussian-weighted mean, minus a small offset:

local mean threshold
foreground⁡(x, y)=I(x, y)>meanwindow(x, y)−C\operatorname{foreground}(x,\,y)=I(x,\,y)>\mathrm{mean}_{\text{window}}(x,\,y)-C

where C is a constant offset that suppresses noise (use a multiplicative form, I < ratio × mean, for dark objects under a gradient as in Lab P2-L2). The window must be larger than the objects you want to keep and small enough that the light is almost constant inside it.

Connected components

Two foreground pixels are connected if a path of foreground pixels joins them, moving to neighbours. With 4-connectivity the neighbours are left, right, up and down; with 8-connectivity the diagonals count too. 4-connectivity separates objects that touch only at a corner; 8-connectivity joins them. Components are found by a flood fill (breadth-first or depth-first search, with a stack or queue, not recursion, for large regions) or a two-pass labelling algorithm. For each region, report:

region properties
area=number of pixels\text{area}=\text{number of pixels}
centroid=(mean x, mean y)\text{centroid}=(\text{mean}\,x,\,\text{mean}\,y)
box=(min⁡ x, min⁡ y, max⁡ x, max⁡ y)\mathrm{box}=(\min \,x,\,\min \,y,\,\max \,x,\,\max \,y)

Morphology

Erosion removes a foreground pixel unless all pixels under a small structuring element (for example a 3 × 3 square) are foreground. It shrinks regions and deletes specks smaller than the element. Dilation adds a pixel if any pixel under the element is foreground; it grows regions and fills small holes. Their combinations are the useful ones:

  • Opening = erosion then dilation: removes small specks and thin bridges and restores the size of what remains.
  • Closing = dilation then erosion: fills small holes and gaps and joins nearby fragments.

Both change shape slightly: corners are rounded, and thin parts of real objects can disappear. The element size is a choice of how small a feature to treat as noise.

A worked example: Otsu on a 16-bin histogram

The histogram in the figure has these counts for the 16 grey-level bins (0 to 15), a total of 164 pixels:

bin counts
2, 9, 22, 30, 21, 9, 3, 1, 1, 2, 4, 9, 15, 20, 12, 42,\,9,\,22,\,30,\,21,\,9,\,3,\,1,\,1,\,2,\,4,\,9,\,15,\,20,\,12,\,4

The table lists the between-class variance for each candidate threshold (pixels at or below bin t are class 0). The weights are fractions of 164.

tw0w1μ0μ1σB²
20.200.801.618.227.03
30.380.622.279.7713.32
40.510.492.7011.2918.42
50.570.432.9212.0820.60
60.590.413.0212.3521.14
70.590.413.0612.4321.22
80.600.403.1112.5021.19
90.610.393.2312.6120.93
100.630.373.4912.7820.04
110.690.314.0913.1017.39

The variance is largest at t = 7 (σB2 = 21.22), in the valley between the two humps: pixels in bins 0 to 7 are floor, bins 8 to 15 are objects. Notice what the numbers say: moving the threshold left into the floor hump gives a small class 0 and a large mixed class 1, and moving it right into the object hump does the opposite; in both cases the classes overlap more and σB2 falls.

Cleaning a mask: the cost

Suppose the mask contains a 9 × 9 object and a 2-pixel speck. An opening with a 3 × 3 square removes the speck, since it cannot contain a 3 × 3 block, and restores the object to its original shape except that its sharp corners may be rounded by a fraction of a pixel. Opening with a 11 × 11 square would delete the object as well. The structuring element is therefore a statement about the smallest real object you expect: choose it below that size and above the noise.

Back to the first lab

In Lab P2-L1, the red mask of the simulated image was followed by connected components with a minimum area of 6 pixels. A speck of 6 or more mask pixels would be reported as an object. An opening with a 3 × 3 element before labelling removes any speck smaller than 3 × 3 more cleanly than a size threshold, and does not depend on region counting. Both approaches are used; the difference is where you put the knowledge of “how small is too small”.

Engineering implications and limitations

Plot the histogram before choosing. Looking at it takes seconds and tells you whether one threshold can exist. A single hump, or three humps, says that thresholding on this channel will not work.

Choose the channel. Threshold the channel in which the classes differ: saturation or hue for a colourful object on a grey floor, value for a dark object, a depth image for objects above a plane. A threshold on the wrong channel can be perfect on one image and useless on the next.

Do not tune by eye on one image. Collect images across the lighting, backgrounds and objects you will meet, choose the threshold or the method on those, and test on others. Prefer methods that adapt (Otsu on each image, local thresholds) when conditions vary.

Put the area filter in physical units. A minimum area of 50 pixels means different things at different distances. If the camera geometry is known, convert a minimum physical size into pixels using the range, or filter after projecting into the world.

Expect merging and splitting. Touching objects form one component and one object with a hole can form two. Components give a first answer; separating touching objects needs more (depth, edges, learned detectors, Module 6).

Mind the speed. Thresholding is trivial; a stack-based flood fill is linear in the number of pixels. Both are cheap enough for every frame, which is one reason classical segmentation remains in use in constrained settings such as conveyor belts, where light and background are controlled.

Common misconceptions

“There is a correct threshold for the scene.”

There is a good threshold for one lighting and one camera setting. Change either and the best value moves.

“Otsu’s method always finds the right threshold.”

It assumes two roughly balanced classes. With a small foreground, uneven light or three classes, it can pick a threshold that splits the wrong way.

“Connected components separate touching objects.”

They separate objects with a gap of background between them. Touching objects of the same class form one region.

“Opening and closing are inverse operations.”

They are not: each removes a different kind of defect (specks or holes), and neither restores what the other removed.

Engineering takeaways

  1. A global threshold cuts the histogram between the classes; Otsu’s method picks the cut that maximises the between-class variance σB² = w0 w1 (μ0 − μ1)².
  2. Uneven light, noise and small foregrounds defeat a single threshold; a local threshold compares each pixel with its own neighbourhood.
  3. Connected components (4- or 8-connected) turn a mask into regions with area, centroid and box; filter by area.
  4. Opening removes specks, closing fills gaps; both round corners and need an element size chosen from the smallest real feature.
  5. Look at the histogram, choose the right channel, test on varied images and express size limits in physical units.

References and further reading

The histogram and the Otsu table were computed for this lesson.

  • Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9(1): the original method.
  • Gonzalez, R. C. and Woods, R. E. (2018). Digital Image Processing, 4th ed. Pearson: thresholding, morphology and connected components.
  • Haralick, R. M. and Shapiro, L. G. (1992). Computer and Robot Vision, Vol. 1. Addison-Wesley: connected component labelling and mathematical morphology.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A histogram has a single hump. Can you use a global threshold, and what would you try?

Explanation

Not reliably: with one hump there is no valley to cut in, so the classes overlap in grey level. Try another channel in which the classes separate (hue or saturation, depth), a local threshold if the cause is uneven light, or edges or texture instead of brightness.

Question 2

A mask has two squares of 5 × 5 that touch at one corner. How many components do 4-connectivity and 8-connectivity find?

Explanation

With 4-connectivity, two (they share no side); with 8-connectivity, one (the corner pixels are diagonal neighbours). The choice should match what “touching” means for your objects.

Question 3

You apply an opening with a 5 × 5 element to remove noise, and a thin 3-pixel-wide cable disappears from the mask. Why, and what are your options?

Explanation

Opening erodes first, and a 3-pixel-wide line cannot contain a 5 × 5 block, so the erosion deletes it and the dilation cannot bring it back. Use a smaller element (for example 3 × 3 or 2 × 2), or remove noise with an area filter or a median filter and keep thin structures.

What’s next

Next: edges and contours: where brightness changes, and how to turn boundaries into simple shape measurements.

Practice this lesson

Detect Colored Objects in an Image

Find coloured objects in an image with a colour-space threshold and locate their centres.

Next lesson

P2-T5 — Edges, Contours and Simple Geometric Features

Compute gradients and edges and measure simple shape properties.

Nothing below is required to finish this lesson.