FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 02

Engineering Guides · Module 02

Sensor combination: camera + IMU

The IMU gives scale and bridges blur; the camera removes the IMU’s drift. Together they are visual-inertial odometry.

  • Advanced
  • Module 02: Sensor Selection, Trade-offs & Integration
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-05

Assumptions: General fusion guidance. Named algorithms appear only where their READMEs were checked (see the SLAM Selection Guide). No performance figures are given.

1. Engineering problem

A single camera has no metric scale and loses tracking when the image blurs or the scene is poor; an IMU alone drifts within seconds. Fusing them gives scaled, high-rate motion with bounded drift.

2. When is this needed?

  • Small robots and drones where weight and power rule out LiDAR.
  • Fast motion where camera-only tracking fails.
  • Indoor and GNSS-denied navigation with textured scenes.

3. Selection and design decisions

What each sensor contributes

SensorContributionWeakness the other covers
CameraDrift-bounding visual features, appearance for recognitionNeeds light and texture, no scale alone, blurs in fast motion
IMUScale (through acceleration), orientation, high-rate prediction between framesIntegrates noise and bias into drift

Fusion approaches

  • Tightly coupled visual-inertial estimators fuse feature tracks and IMU measurements in one estimator: VINS-Fusion (optimisation-based, per its README) and OpenVINS (filter-based, MSCKF, per its README) are examples; see the SLAM Selection Guide for what each README states.
  • Loosely coupled: a visual odometry output fed to an outer filter; simpler, less accurate and less robust to fast motion.

4. Advantages and disadvantages

  • Tight coupling is usually more accurate and more sensitive to calibration and time offset.
  • Rolling shutters need modelling at high speed.
  • More features cost compute.

5. Requirements and constraints

  • Time synchronisation to within what the estimator tolerates (use the Sensor Time Synchronization Simulator).
  • Calibrated extrinsics between every fused pair, with the transform direction documented (see the frame-conventions guide).
  • IMU rate well above the camera rate; IMU noise parameters known.
  • Camera–IMU extrinsics and time offset calibrated, or estimated online (some estimators support it).

6. Architecture

camera (stamp at exposure) ─┐
                            ├─▶ visual-inertial estimator ─▶ pose, velocity, biases
IMU (high rate) ────────────┘

7. Implementation

  1. Calibrate camera intrinsics, then camera–IMU extrinsics and time offset on a target.
  2. Obtain IMU noise parameters (datasheet and a still-log analysis).
  3. Configure the estimator with those values; verify initialisation with a slow, rich-motion start.
  4. Validate against a reference trajectory or a closed loop.

Compare structures in the Sensor Fusion Architecture Builder.

8. Real examples

  • A handheld mapper, a small indoor drone.

9. Failure scenarios

  • Wrong camera–IMU time offset: estimates jerk during rotation.
  • Wrong extrinsics: systematic drift or scale error.
  • Texture-poor scene or exposure failure: tracking lost.
  • IMU saturation in vibration.

10. Troubleshooting

  1. Check time offset first (shared-event test).
  2. Check IMU noise parameters against a still log.
  3. Check exposure and feature counts per frame.
  4. Re-run calibration if the mount changed.

11. Validation

  • Closed-loop return error below your limit over repeated trials.
  • Recovery after a deliberate tracking loss.
  • Behaviour in the darkest and brightest places of the route.

12. Engineering checklist

13. Further references

  • Class-level guidance written from general engineering knowledge; it contains no product specifications. Product-specific numbers must come from the manufacturer’s datasheet for the exact model and operating mode.
  • ROS 2 message types: sensor_msgs (jazzy branch).

Related on FixTheRobot