Perception · Theory · P2
P2-T6 — Image Preprocessing Failures
Most vision failures are decided before the algorithm runs: light, exposure and noise set what the pixels can say.
A colour detector that works all morning fails in the afternoon sun. An edge detector fires on every shadow. These are not rare bugs; they are the normal way classical image processing meets the real world. This lesson catalogues the failures that matter, shows what each one does to the pixel values, gives a quick diagnostic from the histogram and the intermediate images, and lists the remedies, from controlling the light to making the algorithm tolerant of it.
Learning objectives
After this lesson, you should be able to:
- Name the main preprocessing failures (exposure, noise, shading, shadows, glare, blur, colour cast, distortion) and the pixel evidence of each.
- Diagnose an under- or over-exposed image from its histogram and quantify clipping.
- Estimate how light level affects noise using photon shot noise.
- Choose remedies in the right order: control, measure, normalise, then add robustness.
The idea: the pixels are a product of light and surface
An image does not record objects; it records light after it has bounced off them. The value of a pixel is roughly the surface’s reflectance times the illumination, passed through the camera’s exposure and gain, plus noise, rounded to an integer and clipped to a range. Every one of these factors can change from day to day while the object does not.
The failures fall into groups, with distinct fingerprints:
| Failure | What happens to the pixels | Typical symptom downstream | First check |
|---|---|---|---|
| Underexposure | Values crushed near 0; noise dominates | Missed objects, noisy masks | Histogram piles at the left; count pixels at 0 |
| Overexposure | Values clipped at 255; colour and detail lost | Washed-out colour, merged regions | Histogram piles at the right; count pixels at 255 |
| Low-light noise | Random grain, hot pixels | Speckled masks, false edges | Look at a flat region at 100% zoom |
| Uneven light, shadows | Brightness varies across the scene | One side detected, the other not; shadows as objects | Plot a profile across a flat surface |
| Glare, specular highlights | Local saturation, hue lost | Holes in objects, fake blobs | Look for saturated patches that move with the camera |
| Motion blur, defocus | Edges smeared | Weak gradients, missed features | Measure edge width on a sharp boundary |
| Colour cast, white balance | Channels scaled unevenly | Hue thresholds off | Check a grey or white surface: are R, G, B equal? |
| Lens distortion | Straight lines curved, scale varies | Position errors that grow towards the border | Image a straight edge near the frame edge |
Before changing an algorithm, look at the histogram and a few intermediate images. The first wrong picture is the fault.
The mathematics: a pixel model, clipping and shot noise
A simple model of one pixel’s value is
- ρ
- the reflectance of the surface (what we want to infer), between 0 and 1.
- E
- the illumination at the surface: it changes with the lamp, the sun, shadows and the angle.
- t, g
- the exposure time and the gain (sensitivity) of the camera, set by the user or by auto-exposure.
- n
- noise: read-out noise plus photon shot noise.
- clip(·)
- the sensor and converter limit the value to between 0 and vmax (255 for 8 bits). Anything beyond is lost.
The robot wants ρ. It observes v, which depends on ρ, E, t and g, noise and clipping. If E changes by a factor k, v changes by k (until it clips), so a fixed threshold on v is a threshold on ρE. Only quantities that cancel E, such as ratios (hue, saturation) or local comparisons, are safe.
Clipping
If ρgtE exceeds vmax, the pixel is stored as vmax. Two different surfaces, one twice as bright as the other, can both read 255, and no processing can separate them afterwards. The fraction of clipped pixels is a number to monitor. As a rule of thumb, more than a percent or so of pixels at 0 or 255 in the region that matters is a warning.
Noise and light level
Photons arrive randomly. If a pixel collects on average N photons (more precisely, photo-electrons), the count follows a Poisson distribution with variance N, so the shot noise is √N and the signal-to-noise ratio is
A pixel that collects 10 000 electrons has an SNR of 100 (1% noise). In a scene 100 times darker (or with an exposure 100 times shorter) the same pixel collects 100 electrons, and the SNR is 10 (10% noise). Raising the gain amplifies signal and noise together and cannot recover the lost SNR; the remedy is more light, a longer exposure (which risks motion blur) or a larger sensor or aperture. Read-out noise adds to this and dominates in the darkest conditions.
Motion blur
An object moving across the image at u pixels per second during an exposure of t seconds smears over about u t pixels. A conveyor belt moving 0.5 m/s imaged at 2 mm per pixel moves 250 pixels per second; with an exposure of 20 ms the blur is 5 pixels, enough to wipe out the small features of a part. Halving the exposure halves the blur and halves the light.
A worked example: diagnosing a day of failures
A robot detects orange traffic cones by hue and size. It works at noon and fails at three situations. The histogram and one pixel profile help to diagnose each.
Case 1: dusk, the cones are missed
The grey-level histogram piles up in the lowest three bins, with 12% of the pixels at 0. A cone pixel reads (35, 14, 4): value 0.14 and hue computed from channels that differ by a few counts. The detector’s minimum value of 0.2 rejects every cone pixel. The cause is underexposure, and the remedy is exposure (a longer exposure or higher gain with a noise check), or a value limit lowered after confirming that hue is still stable at that level.
Case 2: midday, cones are partly missed
The histogram has a spike in the last bin, with 4% of pixels at 255, and the cone’s sunlit side reads (255, 255, 230), nearly white, saturation 0.1. The side in the sun has clipped. The remedy is a shorter exposure or an aperture stop (for the sunlit region), or an exposure setting from the region of interest rather than from the whole frame, which includes a bright sky.
Case 3: late afternoon, shadows of the cones are detected
A hue range cannot reject a shadow on an orange-brown floor: the shadow keeps the hue of the floor and only lowers the value. With a minimum value of 0.2 the shadow passes and a patch of floor becomes an object. A better test uses saturation and hue together (a shadowed brown floor is low in saturation compared with a cone), and size and shape checks on the resulting region.
How much does the light change the pixel?
The ratio of illumination between full sun (about 100 000 lux) and a heavily overcast sky (about 1 000 lux) is 100:1, and between a well lit office (500 lux) and dusk outdoors (a few lux) also two orders of magnitude (these are typical order-of-magnitude figures). An 8-bit sensor has 256 levels and cannot span a 100:1 or 1000:1 range at once without auto-exposure. That is why exposure control is part of the vision system, not a camera detail, and why a detector tuned at a fixed exposure fails when the exposure changes.
Engineering implications and limitations
Control what you can. The cheapest robust solution is often to fix the lighting: a shroud and LED ring on an inspection station, a fixed exposure and white balance, an infrared illuminator with a matching filter. A system in a controlled cell can use simple methods that would fail outdoors.
Measure and monitor. Log histograms, the fraction of clipped pixels, mean brightness and noise level for every frame or every minute. A vision stack that does not know that its input has degraded cannot respond. Treat an out-of-range histogram as a sensor fault.
Use exposure control deliberately. Auto-exposure follows the average brightness of the frame, which may be dominated by the sky or by a bright wall, and it changes values from frame to frame, which breaks thresholds and tracking. Meter on a region of interest, lock the exposure for tasks that need stable values, or compensate by using the exposure value as an input.
Normalise, then threshold. Divide by an estimate of the illumination, use ratios (hue, saturation, chromaticity), subtract a background model, or threshold locally (Lab P2-L2). Prefer relative tests to absolute ones.
Add redundancy. Combine cues that fail differently: colour with shape and size, intensity with depth, or two cameras with different exposures (a standard approach for high dynamic range). A detector that depends on a single cue fails when that cue does.
Test in the bad cases and design the fallback. Build a library of images from dawn, dusk, rain, glare, dirty lenses and motion, and run it before every release. Decide what the robot does when the input is bad: slow down, stop, use another sensor or ask for help. It is far better to report “I cannot see” than to report a confident wrong answer.
Common misconceptions
“If it works on the test images, preprocessing is done.”
It works for the lighting of those images. Light changes by orders of magnitude in a day, and each change moves the pixel values.
“Raising the gain brings back the detail in a dark image.”
It amplifies the noise equally. The SNR is set by the photons collected; only more light, a longer exposure or a larger aperture improves it.
“Clipped pixels can be fixed in software.”
Once a value is stored as 255 the original is gone. The only fix is to capture the image with less exposure.
“A shadow is not a problem for a hue-based detector.”
Shadows keep the hue and lower the value, so they pass hue tests. They are exactly the case where a value or saturation limit decides.
Engineering takeaways
- Pixel value ≈ clip(gain × time × reflectance × illumination + noise): illumination and exposure change the value without changing the object.
- Diagnose with the histogram (piles and clipped end bins) and by looking at the intermediate images; the first wrong stage is the fault.
- Shot noise gives SNR = √N: 100 times less light is 10 times worse SNR, and gain does not fix it.
- Motion blur is speed × exposure time in pixels; shorter exposure reduces blur and light together.
- Control the light, monitor the inputs, meter and lock the exposure, normalise or use ratios and local tests, add redundant cues, and design a safe fallback.
References and further reading
The illumination levels quoted are standard orders of magnitude; the pixel values in the cone example are illustrative.
- Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: image formation, noise and exposure.
- Janesick, J. R. (2007). Photon Transfer: DN → λ. SPIE Press: photon shot noise and sensor noise models.
- Hasinoff, S. W. et al. (2016). Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (SIGGRAPH Asia): combining short exposures against noise and clipping.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A camera sensor collects 400 photo-electrons per pixel in a dim room. What is the shot-noise SNR, and what does using 4 times the exposure time do to it?
Explanation
SNR = √400 = 20 (5% noise). Four times the exposure collects 1 600 electrons, SNR = 40, so the noise halves. The cost is more motion blur and a risk of clipping bright regions.
Question 2
An image of a bright warehouse door has 8% of its pixels at 255. A teammate proposes to “darken it in software” before thresholding. Why will this not recover the detail?
Explanation
The clipped pixels all hold the same value, 255, although the real brightness differed. Darkening by a factor leaves them all equal, so their differences are gone. Only a shorter exposure (or a lower gain or aperture, or multiple exposures) recovers them.
Question 3
Your detector’s thresholds were tuned with the camera on auto-exposure. After a firmware update that changes the metering, detection gets worse, although the cones look the same on screen. What happened and what would you change?
Explanation
The auto-exposure now settles at different brightness, so pixel values for the same cone changed and the absolute thresholds no longer match. Fix the exposure and white balance for the task, or meter on a region of interest, and prefer ratio-based or local tests that do not depend on the absolute exposure.
What’s next
That completes Module 2. Next: the geometry of the camera itself: how 3D points land on the image, and what the camera constants mean.
Related content
Nothing below is required to finish this lesson.
- BeforeP2-T5: Edges, Contours and Simple Geometric Features
Compute gradients and edges and measure simple shape properties.
- LabP2-L2: Tune Thresholds Under Changing Lighting
Make a detector work across lighting conditions with normalisation and adaptive thresholds.
- LabP2-L3: Diagnose a Failing Image-Processing Pipeline
Find which stage of a filter-threshold-contour pipeline fails and repair it.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.