Skip to the lesson
FixTheRobot Explore Robotics / Localization & SLAM / M6 Visual and Visual-Inertial SLAM / M6-T4

Localization & SLAM · Theory · M6

M6-T4 — Monocular, Stereo, and RGB-D SLAM

The camera type decides where the depth and the scale come from, and therefore how the system fails.

  • Lesson 4 of 9
  • Module 6: Visual and Visual-Inertial SLAM
  • Theory
  • About 9 minutes to read

Chapter M6-T3 showed that a single camera recovers structure and motion only up to scale. Three sensor configurations deal with that differently: a monocular camera accepts the ambiguity, a stereo pair measures depth with a known baseline, and an RGB-D camera measures depth directly with an active sensor. ORB-SLAM2 (Mur-Artal and Tardós, 2017) treats all three in one framework. This lesson compares them: what each measures, what it costs, and how it fails.

Learning objectives

After this lesson, you should be able to:

  • Explain where the scale comes from in monocular, stereo and RGB-D SLAM.
  • Compute stereo depth from disparity and the depth error for a given baseline.
  • Describe the monocular initialisation problem and the scale drift it leaves.
  • Choose a camera configuration for an application from range, lighting and budget constraints.

The problem and the intuition

Your two eyes judge distance by comparing what each sees; close one and you can still move around by using motion, but you cannot tell whether the cup in your hand is a real cup or a model. A monocular camera is the one-eyed observer: it relies on motion, and is blind to absolute size. A stereo camera is two eyes with a fixed, known separation: every frame gives metric depth, but only for points near enough that the two views differ measurably. An RGB-D camera projects its own light (structured light or time of flight) and reads depth directly, but only in the indoor range where its light is stronger than the sun.

No configuration is best everywhere. A drone with a small payload may use a monocular camera plus an IMU; a car uses stereo or lidar; an indoor service robot uses RGB-D. The choice is a decision about which failure you can live with.

Monocular: free scale. Stereo: metric depth that degrades with distance. RGB-D: dense metric depth over a short range, but indoors only.

Stereo depth error grows with the square of depthDepth error of a stereo camera for a half-pixel disparity error and a focal length of 700 pixels: with a 0.12 m baseline the error reaches 1.5 m at 16 m; with a 0.5 m baseline it is much smaller.048121600.40.81.21.6depth Z (m)depth error (m) baseline 0.12 mbaseline 0.50 m δZ = Z² δd / (f·b)δd = 0.5 px, f = 700 px Far points constrain rotation,not translation.
Stereo depth error against depth for two baselines.

The three configurations

Problem. Obtain 3D points and a metric trajectory from a camera system, in real time, with a failure behaviour that is acceptable for the application.

Assumptions. Calibrated intrinsics; for stereo a known, rigid baseline and rectified images; for RGB-D a depth image registered to the colour image; a static, textured scene.

Monocular

One image per time step. The first map is built by triangulating between two frames chosen for sufficient parallax and a good geometric fit (essential matrix for general scenes, homography for planar ones), with the scale fixed arbitrarily. Everything afterwards is expressed in that scale. Three consequences: (1) Initialisation needs translation and may take seconds, and fails if the robot only rotates. (2) Scale drift: scale is not observed after the start, so it drifts, which loop closure must correct with a similarity transform Sim(3) rather than a rigid one (M4-T6). (3) The trajectory cannot be reported in metres without an external reference.

Stereo

Two cameras with baseline b, rectified so that a point appears on the same row in both. The horizontal difference of its column positions, the disparity d = uL − uR, gives depth directly:

stereo depth and its error
Z=f⋅bdZ=\frac{f\cdot b}{d}
δZ≈Z2⋅δdf⋅b\delta Z\approx \frac{Z^{2}\cdot \delta d}{f\cdot b}

The error grows with Z², so accuracy is excellent near the camera and poor far away. ORB-SLAM2 therefore classifies points as close (depth below about 40 baselines), which are triangulated reliably from a single stereo frame and constrain translation, and far, which are used mainly for rotation and are only triangulated over several frames with enough parallax. Stereo needs no special initialisation, gives metric scale, and runs at the frame rate; its costs are two cameras, rigid mounting, precise calibration and synchronisation (M6-T8), and the matching along the epipolar row.

RGB-D

A depth image gives Z for (most) pixels. A feature at (u, v) with depth Z is back-projected directly: X = (u − cx) Z / fx, Y = (v − cy) Z / fy. ORB-SLAM2 turns it into a virtual stereo coordinate uR = u − f·bv / Z with a virtual baseline bv so that the same machinery applies; dense systems (KinectFusion, ElasticFusion, BundleFusion) instead fuse the depth images into a surface model. Limits come from the physics of the sensor: structured light and many time-of-flight sensors are overwhelmed by sunlight, fail on black, shiny and transparent surfaces, have a valid range of about 0.5 to 4–5 m, and produce holes at depth edges.

b
stereo baseline: distance between the camera centres.
d
disparity uL − uR in pixels.
f
focal length in pixels.
close / far
points nearer or farther than about 40 baselines; close points give translation, far points give rotation.

Strengths, limitations, failure modes

ConfigurationScaleStrengthLimitation / failure mode
Monoculararbitrary, driftscheapest, light, works at any rangeinitialisation needs parallax; scale drift; pure rotation and planar scenes degenerate
Stereometric, from baselinemetric at every frame, no special initialisation, works outdoorsdepth error ∝ Z²; baseline limits useful range; needs rigid mount and calibration; textureless regions have no disparity
RGB-Dmetric, from sensordense depth even in textureless regions, simpleshort range, indoor only for most sensors; black, shiny and transparent surfaces; depth noise grows with range

A worked example: stereo depth and the choice of baseline

  1. Depth from disparity. f = 700 px, b = 0.12 m. A feature with disparity d = 10 px has depth Z = 700 × 0.12 / 10 = 8.4 m. With d = 35 px it is 2.4 m.
  2. The error law. Suppose the disparity is accurate to δd = 0.5 px. At Z = 2 m the depth error is 2² × 0.5 / (700 × 0.12) = 0.024 m (1.2%). At Z = 8.4 m it is 8.4² × 0.5 / 84 = 0.42 m (5%). At 16 m it is 1.5 m (9.5%). Doubling the distance quadruples the absolute error.
  3. The baseline. With b = 0.5 m, the error at 8.4 m is 8.4² × 0.5 / (700 × 0.5) = 0.10 m, four times smaller. A wider baseline reaches farther, but near objects may then be seen by only one camera, and the rig is harder to keep rigid.
  4. Close and far. With b = 0.12 m the close/far boundary is 40 × 0.12 = 4.8 m, which corresponds to a disparity of 700 × 0.12 / 4.8 = 17.5 px. A point with disparity 10 px (8.4 m) is a far point: it fixes the camera’s heading but not its position. In an open field, where everything is far, the stereo camera behaves like a (rotation-only) monocular one.
  5. RGB-D virtual disparity. A depth camera with f = 525 px and a virtual baseline of 0.08 m sees a surface at 2 m: uR = u − 525 × 0.08 / 2 = u − 21 px. The same surface at 5 m is 8.4 px, where a 0.5 px quantisation error is 6%; this is one reason RGB-D accuracy falls with distance even before the sensor’s own noise.
  6. Outdoors. A structured-light RGB-D camera in direct sun returns almost no valid pixels: the projected pattern is far dimmer than the ambient light. Stereo works because it needs only ambient light.

Engineering implications and limitations

Engineering implications

  • Pick by range and lighting. Indoors under 5 m: RGB-D is easy. Outdoors or long range: stereo (wider baseline) or lidar. Weight- or cost-limited: monocular plus IMU (M6-T6).
  • Design the baseline for the working distance. A rule of thumb is that useful depth extends to roughly 50 to 100 times the baseline.
  • Calibrate and lock the stereo rig. A vibration that shifts one camera by a fraction of a pixel biases all the depths; check the epipolar row error in the field.
  • Mask invalid depth. Discard RGB-D pixels with zero or out-of-range depth and treat depth edges with suspicion.
  • Synchronise the cameras. A 10 ms offset between stereo images of a moving platform is a systematic disparity error.
  • Report monocular results with a Sim(3) alignment when comparing against ground truth (M4-T6).

Limits

A configuration is a bet on the environment. Stereo in a dark tunnel, RGB-D in a sunlit atrium and a monocular camera over a featureless lake each fail for a physical reason that no tuning fixes. Active stereo (a projected pattern added to a stereo pair) and event cameras address some cases at extra cost. Learned monocular depth networks give plausible depth from one image but with scale that is also guessed and that can be wrong in unfamiliar scenes; they are an input to be checked, not a replacement for geometry.

Common misconceptions

“Stereo gives accurate depth at any distance.”

The error grows with the square of the depth; beyond a few tens of baselines stereo depth is poor.

“RGB-D works everywhere a camera works.”

Most depth sensors need dim ambient light and a reflective surface; sunlight and black or glassy objects defeat them.

“Monocular SLAM just needs more features to get scale.”

Features cannot give scale: it is unobservable from images alone. An extra sensor, or a known object, is required.

“A wide baseline is always better.”

It improves far depth but loses near overlap, needs a stiffer mount and a harder matching problem.

Engineering takeaways

  1. Monocular has an unobservable scale; stereo gets it from the baseline; RGB-D gets it from the depth sensor.
  2. Stereo depth is Z = f·b/d and its error grows as Z² δd/(f·b): good near, poor far.
  3. Close points constrain translation; far points constrain rotation. Choose the baseline for the working range.
  4. RGB-D is dense and simple but limited by range, sunlight and surface reflectance.
  5. The camera choice sets the failure modes: choose for your environment, not only your budget.

References and further reading

Primary papers and authoritative documentation.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A stereo rig has f = 700 px and b = 0.12 m. What is the depth at a disparity of 20 px, and is it a close or far point under the 40-baseline rule?

Explanation

Z = 700 × 0.12 / 20 = 4.2 m. The boundary is 40 × 0.12 = 4.8 m, so 4.2 m is a close point: it can be triangulated from a single frame and constrains translation as well as rotation.

Question 2

Why does a monocular SLAM trajectory drift in scale, while a stereo one does not (to first order)?

Explanation

Monocular scale is unobservable from images: it is fixed at the start and carried through triangulation, so small errors in each step accumulate in the scale. Stereo measures depth with a fixed, known baseline in every frame, so the scale is anchored to the baseline continuously.

Question 3

A warehouse robot must work both inside and in a sunny yard. Why might an RGB-D camera alone be a poor choice?

Explanation

Structured-light and many time-of-flight sensors are overwhelmed by direct sunlight and have a short range, so outdoors they return little valid depth. A stereo camera (or lidar) works in daylight and over longer ranges.

What’s next

With the camera configurations understood, the next lesson looks at a complete keyframe-based visual SLAM system, ORB-SLAM, and how its three threads share one map.

Practice this lesson

Understand Visual Odometry Drift

Run visual odometry on a loop, then see how scale and rotation errors compound and what a loop closure fixes.

Next lesson

M6-T5 — ORB-SLAM Architecture

Explain tracking, local mapping and loop closing threads, keyframes and the covisibility graph.

Nothing below is required to finish this lesson.