Perception · Theory · P3

P3-T5 — Depth Cameras and RGB-D Data

Three ways to measure depth, three ways to fail. Know which one you have, and what its error looks like.

Lesson 5 of 6
Module 3: Geometry and Camera Perception
Theory
About 10 minutes to read

A depth camera returns, for each pixel, a distance. That removes the ambiguity of Lesson P3-T4, but it replaces it with a measurement that has its own physics and failures. Stereo cameras triangulate between two views, structured-light cameras project a pattern, and time-of-flight cameras time light. This lesson explains each, derives the stereo depth and its error, lists the artefacts you will see in the data, and shows how a colour image and a depth image are combined into RGB-D.

Learning objectives

After this lesson, you should be able to:

Explain how stereo, structured light and time of flight measure depth and where each works poorly.
Compute depth from disparity and the depth error as a function of distance, baseline and focal length.
Describe the common artefacts in depth images and how to handle them.
Back-project a depth image into a 3D point cloud and combine it with a colour image.

The idea: three ways to turn light into distance

Cover one eye and hold a finger in front of you; close the eyes alternately: the finger jumps against the background. The nearer the finger, the bigger the jump. That jump between two viewpoints is the disparity, and measuring it is how a stereo camera finds depth.

Two cameras a baseline b apart see the same point at different columns. The difference is the disparity, which is large for near points and small for far points.

Depth sensing technologies compared
Technology	How it measures	Strengths	Typical weaknesses
Passive stereo	Matches pixels between two cameras and triangulates	Works outdoors in daylight; no emitter; cheap; high resolution	Needs texture; fails on blank walls; error grows as z²; matching errors on repeated patterns and reflections
Active stereo / structured light	Projects an infrared pattern and matches it (stereo or with a known pattern)	Works on blank surfaces; dense depth indoors	Washed out by sunlight; limited range; multiple cameras interfere; poor on dark or shiny surfaces
Time of flight (ToF)	Measures the delay or phase of emitted light for each pixel	Direct depth; compact; no texture needed; constant error with range for a given reflectivity	Multipath (corners read too far); flying pixels at edges; sunlight; low resolution; motion artefacts
LiDAR (for comparison)	Scans a laser beam and times the return	Long range, precise, works in daylight	Sparse, costly, moving parts or solid-state limits; glass and dark surfaces

Every depth sensor reports a number for every pixel it can, and nothing for the rest. A missing depth is not a distance of zero.

The mathematics: stereo, time of flight and RGB-D

Stereo depth from disparity

For a rectified stereo pair (the two images are aligned so that matching points lie in the same row) with focal length f in pixels and baseline b, a point at depth Z appears at column u_L in the left image and u_R in the right. By similar triangles

stereo disparity and depth

d=u_{L}-u_{R}=\frac{f\cdot b}{Z}

Z=\frac{f\cdot b}{d}

d: disparity in pixels: the column difference of the same point in the two images. It is positive, large for near points and small for far ones.
f: focal length in pixels (the same for both rectified cameras).
b: baseline: the distance between the two optical centres, in metres.
Z: depth along the optical axis, in metres.

Depth error

Differentiate Z = f b / d with respect to d:

stereo depth error

\frac{\mathrm{dZ}}{\mathrm{dd}}=-\frac{f\,b}{d^{2}}=-\frac{Z^{2}}{f\,b}

\text{so}\,\sigma _{Z}\approx \frac{Z^{2}\,\sigma _{d}}{f\,b}

where σ_d is the error of the disparity (often 0.1 to 0.5 pixel for a good matcher). The depth error grows with the square of the distance and shrinks with a longer baseline and a longer focal length. This is the quantity you explored in Lab P3-L3.

Time of flight

A ToF sensor emits light and measures the round-trip time (or, for continuous-wave sensors, the phase shift of a modulated signal). With the speed of light c,

time-of-flight depth

Z=\frac{c\cdot \Delta t}{2}

Z=\frac{c\cdot \Delta \phi }{(4\pi \,f_{\bmod})\text{(continuous wave)}}

Light covers 3 m out and back in 20 ns, and a depth resolution of 1 cm needs timing to about 67 ps, so sensors measure phase rather than time directly. Continuous-wave ToF has an ambiguity range c/(2 f_mod): beyond it the phase wraps and distances alias (at 20 MHz about 7.5 m).

From a depth image to points and RGB-D

A depth image is a map Z(u, v). Each valid pixel is back-projected with the intrinsics of the depth camera (Lesson P3-T3):

back-projection of a depth pixel

X=\frac{(u-c_{x})\,Z}{f_{x}}

Y=\frac{(v-c_{y})\,Z}{f_{y}}

Z=Z(u,\,v)

To attach colour, each 3D point is transformed into the colour camera’s frame with the extrinsics between the two cameras and projected with the colour camera’s intrinsics to pick up a colour. This is registration, and it fails if the extrinsics are wrong: the colour is then painted on the wrong points, most visibly at depth edges. Many RGB-D cameras register internally and deliver an aligned pair.

Artefacts to expect

Invalid pixels (reported as 0 or NaN): no match, too dark, too shiny, out of range, transparent, occluded in one stereo view. Never treat them as zero distance.
Depth discontinuity errors: at an object’s edge the depth of the object and the background are mixed; ToF and structured-light sensors produce “flying pixels” floating between the two.
Occlusion shadows: the region behind an object that one stereo camera or the projector cannot see has no depth, so objects seem to have a halo of invalid pixels on one side.
Systematic errors: surface-dependent bias (ToF reads dark materials too far; shiny surfaces mirror other objects), multipath (corners and concave shapes).
Quantisation: depth steps get coarser with distance for stereo, since disparity is quantised in pixels or fractions of a pixel.

A worked example: a stereo camera at five distances

A stereo camera has f = 600 pixels and baseline b = 12 cm, with a disparity error σ_d = 0.25 pixel.

Stereo depth, disparity and error at six distances
Depth Z	Disparity d = f b / Z	Depth error σ_Z = Z² σ_d / (f b)	Relative error
1 m	72.0 px	0.3 cm	0.3%
2 m	36.0 px	1.4 cm	0.7%
3 m	24.0 px	3.1 cm	1.0%
4 m	18.0 px	5.6 cm	1.4%
6 m	12.0 px	12.5 cm	2.1%
8 m	9.0 px	22.2 cm	2.8%

At 1 m the camera gives depth to about 0.3 cm. At 4 m the error is 6 cm, and at 8 m 22 cm (2.8%). The disparity at 8 m is only 9 pixels, so a quarter-pixel error is a 2.8% error in the disparity. For a robot that must place an object within 2 cm at 4 m this camera is not good enough; increasing the baseline to 0.4 m would reduce the error at 4 m to 1.7 cm, the figure used in the lab.

Using the depth image

The camera reports a depth image of 640 × 480 pixels. About 5% of pixels are invalid (0), mostly at the left edges of objects, where one camera cannot see. A back-projection that skips invalid pixels produces about 292 thousand 3D points per frame; one that treats the 0 as a depth produces 15 thousand points on the camera itself, which any later obstacle check will happily report as an obstacle 0 m ahead. (Lab P3-L3 has this exact bug in its starter.)

Choosing a sensor

For a robot working indoors with blank walls and moderate range, active stereo or structured light is usually a good fit. For a robot used outdoors in daylight, passive stereo or LiDAR works and structured light does not. For close-range manipulation, ToF or structured light give dense depth quickly. For navigation at 20 to 100 m, a long baseline stereo rig or LiDAR is needed. Each answer follows from the error formula above and from the failure modes in the table.

Engineering implications and limitations

Quote accuracy at a distance. A sensor’s “millimetre accuracy” is for a particular range and surface. Ask for the error as a function of range on the surfaces you have, and test it.

Handle invalid pixels explicitly. Keep a validity mask. Do not fill invalid pixels with 0 or the maximum range in a way that later stages mistake for a distance or free space. If hole-filling is needed for a dense output, mark the filled pixels as such.

Filter flying pixels and edges. Remove points whose depth differs strongly from neighbours on one side, or whose normal is nearly perpendicular to the viewing ray. Erode the valid mask by a pixel at depth edges before measuring.

Calibrate the whole system. Stereo needs good intrinsics and extrinsics for rectification; a few tenths of a pixel of rectification error costs as much accuracy as the matcher’s error. RGB-D registration needs the depth-to-colour extrinsics. Recheck them after knocks.

Mind sunlight and surfaces. Test outdoors in direct sun, on black surfaces, on glass, mirrors, wet floors, and with other cameras running. Specify what the robot does when depth is missing over large areas.

Choose the representation. Depth images keep the neighbourhood structure and are cheap to process (ground removal, edges), while point clouds are convenient for 3D geometry. Convert once and attach the sensor frame and timestamp (Lesson P1-T4, Module 5).

Account for latency. Many depth cameras have tens of milliseconds of latency from exposure to output. Use the exposure timestamp, not the arrival time, when fusing with other sensors (Module 8).

Common misconceptions

“A depth camera gives the true distance at every pixel.”

It gives an estimate with a range- and surface-dependent error, and nothing at all for the pixels where measurement failed.

“Stereo accuracy is the same at all distances.”

The depth error grows as Z² (for a constant disparity error), so a camera that is good at 1 m can be poor at 8 m.

“A missing depth value means the point is at distance zero or at the maximum range.”

It means the sensor could not measure it. The object may be dark, shiny, transparent, occluded in one view, or out of range.

“Time of flight does not depend on the surface.”

The measured time or phase depends on the reflected signal; dark surfaces, multipath and semi-transparent materials produce systematic errors.

Engineering takeaways

Stereo uses Z = f b / d with a depth error Z² σ_d / (f b); structured light projects a pattern; time of flight uses Z = c Δt / 2 or the phase shift.
Each technology has typical failures: texture and z² error (stereo), sunlight (structured light), multipath and flying pixels (ToF).
Depth images contain invalid pixels and edge artefacts: carry a validity mask and never treat a missing value as a distance.
Back-project a depth pixel with (u − c_x) Z / f_x, (v − c_y) Z / f_y, Z; register colour with the depth-to-colour extrinsics.
Choose the sensor from the range, surface and light conditions, quote accuracy at a distance, and test the failure cases.

References and further reading

The stereo table was computed for this lesson from the formulas given.

Scharstein, D. and Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47: stereo matching and its evaluation.
Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: epipolar geometry and triangulation.
Hansard, M., Lee, S., Choi, O. and Horaud, R. (2013). Time-of-Flight Cameras: Principles, Methods and Applications. Springer: continuous-wave ToF, ambiguity range and error sources.
Intel RealSense and Microsoft Azure Kinect documentation: practical descriptions of active stereo and ToF sensors, their invalid pixels and noise behaviour.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A stereo rig has f = 800 pixels and b = 0.10 m. What is the disparity of a point at 4 m, and what depth does a disparity of 10 pixels give?

Explanation

d = f b / Z = 800 × 0.10 / 4 = 20 pixels. For d = 10 pixels, Z = f b / d = 800 × 0.10 / 10 = 8 m.

Question 2

You double the baseline of a stereo rig and halve the distance. By what factor does the depth error change?

Explanation

σ_Z ∝ Z²/b. Halving Z divides Z² by 4 and doubling b divides by 2, so the error falls by a factor of 8.

Question 3

A robot’s obstacle check reports an obstacle exactly 0 m ahead whenever a depth camera is pointed at a black glossy table. What is the likely cause and the fix?

Explanation

The camera returns 0 for pixels it could not measure on the black surface, and the obstacle code treats 0 as a distance. Keep a validity mask, ignore invalid pixels in distance computations, and handle “mostly invalid” explicitly (unknown, not free and not 0 m).

What’s next

Next: how to measure how wrong a projection is: reprojection error, what it can and cannot tell you, and how it diagnoses calibration.

Practice this lesson

Recover Geometry from Depth Observations

Back-project a depth image and measure distances, heights and widths.

Next lesson

P3-T6 — Projection and Reprojection Errors

Compute reprojection error and diagnose calibration and model errors from it.

Nothing below is required to finish this lesson.

Before
P3-T4: Perspective and Depth Ambiguity
Explain why one image cannot give depth and how scale, size and motion relate.
Lab
P3-L3: Recover Geometry from Depth Observations
Back-project a depth image and measure distances, heights and widths.

Learning objectives

The idea: three ways to turn light into distance

The mathematics: stereo, time of flight and RGB-D

Stereo depth from disparity

Depth error

Time of flight

From a depth image to points and RGB-D

Artefacts to expect

A worked example: a stereo camera at five distances

Using the depth image

Choosing a sensor

Engineering implications and limitations

Common misconceptions

“A depth camera gives the true distance at every pixel.”

“Stereo accuracy is the same at all distances.”

“A missing depth value means the point is at distance zero or at the maximum range.”

“Time of flight does not depend on the surface.”

Engineering takeaways

References and further reading

Knowledge check

Question 1

Question 2

Question 3

What’s next

Recover Geometry from Depth Observations

P3-T6 — Projection and Reprojection Errors

Related content

Get notified when the next lesson and new labs are released.