M6-T3 — Visual Odometry

Learning objectives

After this lesson, you should be able to:

• State the epipolar constraint and say what the essential matrix contains.
• Describe the monocular VO pipeline: essential matrix, triangulation, PnP, local refinement.
• Explain why the translation is recovered only up to scale and what the consequence is for maps and trajectories.
• Identify the degenerate cases: pure rotation, planar scenes and small baselines.

The problem and the intuition

Close one eye, look at a doorway, and slide your head sideways. Nearby objects shift across your view a lot; distant ones hardly move. That parallax contains the motion and the depth, mixed together. If you know how far you moved, you know the depth of everything; if you know the depth of something, you know how far you moved. A single camera knows neither, so it can recover the shape of the motion and of the scene but not their absolute size. A toy train seen through a camera is indistinguishable from a real train at a larger distance.

What the geometry does give is a strong constraint on where a point from image 1 can appear in image 2: on a single line (the epipolar line), because the 3D point must lie on the viewing ray. That constraint, written as an equation, is what we solve for the motion.

Two views give rotation and the direction of translation. The length of the translation, and so the scale of the whole map, must come from somewhere else: a stereo baseline, an IMU, a known object, or a depth sensor.

The algorithm: epipolar geometry, triangulation, PnP

Problem. Given matched points in consecutive images and the intrinsics K, estimate the relative rotation R and translation t of the camera, and then keep estimating the camera pose as it moves.

Assumptions. A calibrated camera (K known, distortion removed); a rigid, static scene; enough parallax (translation, not pure rotation); at least a few correspondences that are correct after outlier rejection (M6-T2); the scene is not entirely planar for the general methods.

1. Normalise. Convert each pixel to a normalised ray x = K⁻¹ [u, v, 1]ᵀ.
2. Epipolar constraint. For a point X seen as x in image 1 and x′ in image 2, the two rays and the baseline are coplanar, so x′ᵀ E x = 0 with the essential matrix E = [t]_× R, where [t]_× is the skew-symmetric matrix of t. E has five degrees of freedom (three for R, two for the direction of t).
3. Estimate E. The five-point algorithm of Nistér (2004) needs 5 correspondences and is used inside RANSAC (M6-T2). The 8-point algorithm (Hartley, 1997) uses 8 and estimates the more general fundamental matrix F = K⁻ᵀ E K⁻¹ for an uncalibrated camera.
4. Decompose. The SVD of E gives two rotations and two translation signs, four combinations. The right one places the triangulated points in front of both cameras (the cheirality check).
5. Triangulate. With R and t (t scaled arbitrarily, for example to unit length), find each point as the intersection of its two rays, by least squares. This gives a first map in an arbitrary scale.
6. Track by PnP. For the next frame, match its features to the triangulated points and estimate the camera pose by solving perspective-n-point (P3P with RANSAC, or EPnP, Lepetit et al., 2009), minimising the reprojection error. Needs 3 points minimum (4 to disambiguate) and keeps the scale of the map.
7. Refine. Minimise the reprojection error over the recent keyframes and points (local bundle adjustment, M4-T7).

E, F

essential and fundamental matrix: encode the relative pose (and, for F, the intrinsics).

[t]_×

the 3 × 3 matrix such that [t]_× v = t × v.

x, x′

normalised image coordinates of a point in the first and second image.

π

perspective division: (X, Y, Z) → (X/Z, Y/Z).

The scale problem and the drift

E is unchanged if t is multiplied by any positive number, so the translation length is unobservable. A monocular VO system fixes the scale arbitrarily in its first step, say a baseline of 1, and then propagates it through the triangulated map. Errors in the scale of each new step compound: a per-frame scale error of 0.2% leads, after 1 000 m of travel, to about 2 m of error even with perfect rotation. Rotation errors are worse, because they tilt the whole subsequent path.

Strengths, limitations, failure modes

A worked example: sideways motion, a depth, and the effect of scale

1. Epipolar lines. The camera translates sideways by t = (1, 0, 0) with no rotation, so E = [t]_× = [[0, 0, 0], [0, 0, −1], [0, 1, 0]]. For x = (0.2, 0.1, 1), Ex = (0, −1, 0.1). The constraint x′ᵀ E x = 0 with x′ = (x′, y′, 1) gives −y′ + 0.1 = 0, so y′ = 0.1: the match lies on the same image row, anywhere along it. Pure sideways motion gives horizontal epipolar lines, which is why a stereo rig can search along a row.
2. Depth from the match. If the match is found at x′ = 0.1, the horizontal disparity is 0.2 − 0.1 = 0.1 (normalised units). With a baseline of 1 unit, depth Z = baseline / disparity = 1 / 0.1 = 10 units. With a baseline of 2 units the same pixels give Z = 20. The image cannot tell which baseline is the true one: that is the scale ambiguity in one line.
3. Fixing the scale. If we know that the point is a door 2 m tall that spans 0.1 normalised units at depth Z, then 2 / Z = 0.1 gives Z = 20 and a baseline of 2 m. Any metric reference (a stereo baseline, an IMU, a wheel odometer, a known object) fixes the scale for the whole map.
4. Conditioning. With a baseline of 0.1 m and a point 20 m away, the parallax angle is atan(0.1 / 20) ≈ 0.29°. A one-pixel error (about 0.1° at f = 600) is a third of the parallax, so the depth is off by 30% or more. Waiting until the camera has moved further (choosing keyframes by parallax, M6-T5) is how systems avoid this.
5. Drift. Suppose every keyframe pair has a scale error of 0.2% and the rotation error is 0.02° per keyframe. After 1 000 m of travel in 500 steps the scale error contributes about 2 m, and the heading error of 500 × 0.02° = 10° tilts the final 200 m of path by about 35 m sideways. This is why visual odometry needs loop closure or other sensors, and why the second lab starts from this drift.

Engineering implications and limitations

Engineering implications

• Initialise with parallax. Wait until the median parallax between the first frame and the current one exceeds a few pixels before triangulating; reject initialisation from pure rotation.
• Choose the model by the scene. Estimate both a homography and an essential matrix during initialisation (as ORB-SLAM does) and pick the one with the lower error; planar scenes break E.
• Track against a map, not only the last frame. PnP against the local map keeps drift small compared with chaining frame-to-frame relative poses.
• Get the scale from a sensor. A stereo pair, an IMU (M6-T6), a depth sensor (M6-T4) or wheel odometry each remove the scale ambiguity. Without them, report trajectories as up to scale and align them with a Sim(3) fit before evaluation (M4-T6).
• Monitor the number of inliers. Fewer than about 30 inlier points after PnP is a warning of imminent tracking loss.
• Mind the rotation. Rotation error is the main contributor to long-range drift; a gyroscope is the cheapest remedy.

Limits

Feature-based VO as described assumes distinct, trackable points. Direct methods (DSO, SVO) minimise photometric error instead and can work in low-texture scenes, but depend on brightness constancy and good photometric calibration. All monocular VO leaves a free scale (and a scale drift). The geometry also assumes a static scene: a large moving object, such as a truck filling the view, produces a confident wrong motion.

Common misconceptions

Engineering takeaways

1. The epipolar constraint x′ᵀ E x = 0 with E = [t]_× R links matched points to the relative pose.
2. Five points suffice for E; RANSAC and the cheirality check pick the right solution.
3. PnP against the triangulated map places new frames and keeps the scale of the map.
4. Monocular VO has an unobservable scale that drifts; stereo, depth, an IMU or wheels fix it.
5. Pure rotation, planar scenes and small baselines are degenerate or ill-conditioned.

References and further reading

Primary papers and authoritative documentation.

• Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press.
• Nistér, D. (2004). An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(6).
• Hartley, R. I. (1997). In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(6).
• Lepetit, V., Moreno-Noguer, F. and Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision 81(2).
• Scaramuzza, D. and Fraundorfer, F. (2011). Visual odometry: Part I: The first 30 years and fundamentals. IEEE Robotics & Automation Magazine 18(4).
• Engel, J., Koltun, V. and Cremers, D. (2018). Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(3).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Visual odometry works with a single camera, but the scale and the depth are only as good as the extra information available. The next lesson compares what monocular, stereo and RGB-D cameras provide and what each costs.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T2: Feature Detection and Tracking

  Compare detect-and-match with optical-flow tracking and the checks that reject bad matches.

• Lab
  M6-L2: Understand Visual Odometry Drift

  Run visual odometry on a loop, then see how scale and rotation errors compound and what a loop closure fixes.