Skip to the lesson
FixTheRobot Explore Robotics / Localization & SLAM / M6 Visual and Visual-Inertial SLAM / M6-T6

Localization & SLAM · Theory · M6

M6-T6 — Visual-Inertial Odometry

The IMU supplies scale, gravity and the fast motion; the camera supplies the drift correction that the IMU cannot.

  • Lesson 6 of 9
  • Module 6: Visual and Visual-Inertial SLAM
  • Theory
  • About 9 minutes to read

M6-T3 and M6-T4 left a monocular camera with an unknown scale and a fragile front end. An inertial measurement unit (IMU) is small, cheap, runs at hundreds of hertz and measures exactly what the camera lacks: acceleration in metres per second squared and rotation rate. Fusing the two, visual-inertial odometry (VIO), is now the standard way to localise drones, phones and headsets. This lesson explains why the pair works so well, what it can and cannot observe, and how the fusion is set up.

Learning objectives

After this lesson, you should be able to:

  • Explain what each sensor contributes and which weakness of the other it covers.
  • Describe IMU preintegration and why it is needed in an optimisation or filter running at camera rate.
  • List what is observable in VIO (scale, gravity direction, velocity, biases) and what is not (global position and yaw).
  • Describe visual-inertial initialisation and the motion it requires.

The problem and the intuition

Close your eyes and walk across a room: you know roughly how far you went and how you turned, because your inner ear and muscles integrate motion. After a minute you would be a metre off. Open your eyes: a glance at the door corrects the error instantly, but you cannot tell from sight alone how many metres it is without already knowing how big things are. The inner ear gives metric motion and drifts; vision gives absolute references and no scale. Together they are far better than either.

The IMU measures proper acceleration a (including gravity) and angular velocity ω in the body frame. Integrating ω once gives orientation; integrating a twice gives position, but errors grow rapidly: a constant accelerometer bias of 0.05 m/s² produces 0.5 × 0.05 × 10² = 2.5 m of error after 10 s. The camera, observing static points, can detect and cancel such drift. In turn, the IMU bridges the intervals in which the camera sees nothing useful (motion blur, a blank wall for a second) and fixes the metric scale.

Vision observes the geometry up to scale and has no drift against the scene; the IMU observes motion in metric units and gravity but drifts. Fusion cancels each one’s weakness.

Camera and IMU complement each otherA timeline of the two sensors. The IMU samples at about 200 hertz and gives continuous but drifting motion. The camera samples at about 20 hertz and gives drift-free but intermittent and scale-free constraints. Preintegration summarises the IMU samples between camera frames.IMU ~200 Hz camera ~20 Hz preintegrated IMU factor between consecutive frames IMU: scale, gravity, fast motion. Camera: drift correction, bias observability.
The two sensors on one timeline, linked by preintegration.

The algorithm: IMU model, preintegration, fusion and initialisation

Problem. Estimate the pose, velocity and IMU biases over time from camera images and IMU readings, in metric scale, at the camera rate.

Assumptions. The camera–IMU extrinsic transform and the time offset between them are known or estimated (M6-T8); the IMU noise follows a white-noise plus bias-random-walk model; the gravity magnitude is known; the scene is static and visually trackable.

The IMU model

IMU measurement model
ω~=ω+bg+ng\tilde{\omega}=\omega +b_{g}+n_{g}
a~=R⊤(aworld−g)+ba+na\tilde{a}=R^{\top}(a_{\text{world}}-g)+b_{a}+n_{a}

The gyroscope reading ω̃ is the true rate plus a slowly varying bias bg and white noise. The accelerometer reading ã is the specific force: it contains gravity, so a stationary IMU reads +9.81 m/s² along the up axis. That is useful: the direction of the reading at rest tells the tilt (roll and pitch) of the sensor.

Preintegration

An IMU produces about ten samples between two camera frames. Integrating them with the current estimate would force a re-integration every time the optimiser changed the starting state. Preintegration (Lupton and Sukkarieh, 2012; Forster et al., 2017) instead integrates the samples in the frame of the first keyframe into relative increments ΔR, Δv, Δp that depend only on the measurements and the biases, together with their covariance and the Jacobians with respect to the biases. A bias change is handled by a first-order correction, so the increments are rarely recomputed. The result is one compact factor between two keyframes, as in M5-T8.

Fusion approaches

  • Loosely coupled. A visual odometry system produces a pose and an EKF fuses it with the IMU. Simple, but the vision does not benefit from the IMU and feature-level errors are lost.
  • Tightly coupled. Raw feature observations and raw IMU enter one estimator, as an optimisation over a window of keyframes (OKVIS, VINS-Mono) or a filter (MSCKF, OpenVINS). More accurate and robust, and the standard choice (M6-T7).

What is observable

With a camera and an IMU under sufficient excitation, the following are observable: the metric scale, the gravity direction (hence roll and pitch), velocity, and the accelerometer and gyroscope biases. Four degrees of freedom are not: the global position (x, y, z) and the yaw about gravity, because neither the images nor the inertial readings change if the whole trajectory is moved or turned about the vertical. Drift of those four quantities remains and only loop closure or an absolute sensor (GPS, a map) removes it (Hesch et al., 2014; Kelly and Sukhatme, 2011).

Initialisation

The estimator needs a good starting point for scale, gravity, velocity and gyroscope bias. A typical procedure (VINS-Mono, ORB-SLAM3): run vision-only SfM on a few frames to get an up-to-scale trajectory; estimate the gyroscope bias by aligning visual and IMU rotations; then solve a linear problem for the scale s, the gravity vector and the velocities by matching IMU preintegrated positions and velocities to the scaled visual ones. This only works if the motion has some acceleration; constant-velocity motion leaves the scale and accelerometer bias unobservable.

bg, ba
gyroscope and accelerometer biases, modelled as random walks and estimated.
ΔR, Δv, Δp
preintegrated rotation, velocity and position increments between keyframes.
g
gravity vector in the world frame; its direction defines the vertical.
s
metric scale factor between the visual trajectory and metres.

Strengths, limitations, failure modes

StrengthLimitation / failure mode
Metric scale and gravity from the IMU; no scale driftNeeds acceleration and rotation to initialise; a robot cruising in a straight line at constant speed cannot be initialised
Handles fast motion and blur: the IMU bridges the gapIMU errors grow quickly: after a long visual outage the pose drifts away (a few seconds for a consumer IMU)
Biases are estimated, not assumedA wrong extrinsic or time offset yields a spurious bias and silent drift (M6-T8)
Cheap, small, widely availableGlobal position and yaw remain unobservable and drift

A worked example: gravity, scale and bias

  1. Gravity gives tilt. A stationary IMU reads ã = (1.70, 0, 9.66) m/s². The magnitude is √(1.70² + 9.66²) = 9.81, as it should be. The tilt from vertical is atan2(1.70, 9.66) = 9.98° ≈ 10°: the sensor is pitched by 10°. This is why roll and pitch are observable and need no vision.
  2. Scale from acceleration. The camera moves with a constant acceleration of 2 m/s² for 2 s, starting at rest. The IMU says the displacement is ½ × 2 × 2² = 4 m. The up-to-scale visual trajectory says 0.8 units. The scale is 4 / 0.8 = 5 metres per unit, and the monocular map can now be reported in metres.
  3. Why constant velocity fails. If the camera moves at constant velocity, the accelerometer reads only gravity and the bias: no acceleration is available to compare with the visual displacement, so the scale cannot be fixed. This is why initialisation procedures ask for a small “wiggle”, and why a car cruising on a straight road may lose scale until it brakes.
  4. Bias drift without vision. An accelerometer bias of 0.05 m/s², left unestimated, gives ½ × 0.05 × t² of position error: 0.5 m at 4.5 s, 2.5 m at 10 s, 22 m at 30 s. The camera constraints reveal and correct the bias within seconds of motion; that is why VIO stays bounded while pure inertial navigation does not.
  5. Preintegration cost. A window of 10 keyframes at 20 Hz with an IMU at 200 Hz covers 0.5 s, so 100 IMU samples. Without preintegration each optimiser iteration over the window would re-integrate all 100 samples; with it, it evaluates 9 compact factors. The saving is what makes optimisation at camera rate possible.

Engineering implications and limitations

Engineering implications

  • Choose the IMU for its noise and bias stability, not its price: a consumer MEMS IMU may drift in seconds; a tactical-grade one in minutes.
  • Set the noise parameters from the datasheet and an Allan-variance test. Copying values from another IMU makes the filter over- or under-confident.
  • Mount the IMU rigidly, close to the camera, and isolate it from vibration; a loose mount adds motion the camera does not see.
  • Calibrate the extrinsic and the time offset (M6-T8) and verify them by checking that the estimated biases stay small and slowly changing.
  • Provide an initialisation motion in the user procedure: a smooth start with rotation and acceleration.
  • Add absolute references (GPS, a pre-built map, loop closure) for long missions: yaw and position drift remain.

Limits

VIO is only as good as the sensor models. The Gaussian white-plus-random-walk IMU model is approximate: real IMUs have temperature-dependent biases, scale-factor errors and axis misalignment. Under sustained constant-velocity motion, or when the gravity direction is poorly excited, parts of the state are weakly observable, and the estimator’s covariance becomes optimistic (the consistency issue discussed in M6-T7).

Common misconceptions

“The IMU removes the need for good vision.”

It drifts quickly. Vision provides the corrections that bound error, and without good vision the bias is not observable.

“VIO can estimate global position and heading.”

Global x, y, z and yaw are unobservable; only scale, gravity direction, velocity and biases are. Those four degrees of freedom drift.

“Initialisation can be done while standing still.”

At rest, scale and the accelerometer bias are unobservable; some acceleration and rotation are needed.

“Loosely coupled fusion is as good as tight coupling.”

It is simpler, but ignores the correlations between the IMU and feature errors and cannot use partial visual information when tracking is poor.

Engineering takeaways

  1. The IMU provides metric scale, gravity (roll and pitch) and short-term motion; the camera provides drift correction and bias observability.
  2. Preintegration folds the IMU samples between frames into one factor that need not be recomputed when the state changes.
  3. Global position and yaw are unobservable; VIO drifts in those four degrees of freedom.
  4. Initialisation needs acceleration and rotation; constant velocity leaves scale unobservable.
  5. Tight coupling and calibrated extrinsics and time offsets are what make VIO accurate.

References and further reading

Primary papers and authoritative documentation.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A stationary IMU reads (0, 0, 9.81) m/s². Another reads (3.4, 0, 9.2). What does the second tell you about the sensor?

Explanation

The magnitude is √(3.4² + 9.2²) = 9.81, so it is also at rest, but the gravity vector is tilted: atan2(3.4, 9.2) ≈ 20° from vertical. The accelerometer at rest gives roll and pitch directly.

Question 2

Why can a monocular VIO system not be initialised while the platform moves at constant velocity?

Explanation

With no acceleration the accelerometer reads only gravity plus bias, so there is no measured metric displacement to compare with the up-to-scale visual displacement. The scale and the accelerometer bias cannot be separated, so the system needs some acceleration and rotation.

Question 3

Which degrees of freedom of the trajectory remain unobservable in VIO, and what removes their drift?

Explanation

The global position (x, y, z) and the yaw about gravity: moving or turning the whole trajectory about the vertical changes neither images nor IMU readings. Loop closure, GPS, or a map prior removes their drift.

What’s next

Knowing what an IMU adds, the next lesson compares how complete visual-inertial SLAM systems are built, as filters (MSCKF) and as sliding-window optimisers (OKVIS, VINS-Mono) with loop closure.

Practice this lesson

Diagnose Camera–IMU Synchronization Errors

Estimate the camera-IMU time offset from angular-velocity signals and see what an uncorrected offset does.

Next lesson

M6-T7 — Visual-Inertial SLAM

Compare filter-based (MSCKF) and optimisation-based (VINS-Mono, OKVIS, Kimera-VIO) visual-inertial SLAM.

Nothing below is required to finish this lesson. The last 2 entries are lessons of other courses on this site that teach the background or go further.