M6-T9 — Failure Modes in Visual SLAM

Learning objectives

After this lesson, you should be able to:

• List the main failure modes of visual and visual-inertial SLAM and the physical or geometric cause of each.
• Compute motion blur in pixels from angular velocity, exposure and focal length.
• Identify the signal in the logs (inlier count, parallax, residuals, covariance) that reveals each failure.
• Describe relocalisation and why it can fail or give a false recovery.

The problem and the intuition

A visual SLAM system is a chain: image quality, then features, then matches, then geometry, then optimisation. The quality of each link limits the next, and the failures are best organised by the link that breaks. Dark, blurred or textureless images give few features. Few features, or ones that are repeated, give few or wrong matches. Wrong matches or degenerate motion give a poorly conditioned or incorrect pose. A wrong pose, fed back into map building, contaminates the map and the following frames.

The most useful habit is to look upstream: when the pose is wrong, ask which earlier link first degraded. Failures are rarely sudden; the count of tracked features falls, parallax shrinks, residuals rise, and only then does the system report “lost”.

Failures are chains. Find the first link that degraded, usually in the logs before the system itself notices.

The catalogue: causes, signals, mitigations

Problem. Recognise why a visual or visual-inertial SLAM run failed or degraded, from sensor, feature, geometry and estimator signals.

Assumptions. The system is a feature-based tracker with keyframes and a map (M6-T5), optionally with an IMU (M6-T6). The analysis uses available logs: tracked feature counts, inlier counts, reprojection residuals, parallax, state transitions, bias estimates and covariances.

Image-level failures

• Motion blur. Blur length in pixels ≈ f · ω · t_exposure for rotation. Blurred gradients make corners disappear and descriptors unrepeatable.
• Low texture. White walls, floors, a clear sky: few corners, G small (M6-T2). Direct and learned methods or active depth sensors can help.
• Illumination change and auto-exposure. Brightness constancy breaks; features saturate or vanish when the camera turns toward a window. Fix the exposure or use photometric calibration.
• Lens flare, rain, dirt. Local loss of image; the features detected there are spurious.

Correspondence failures

• Repeated texture and perceptual aliasing. Identical windows, tiles, aisles: matches are plausible and wrong; a false loop closure folds the map (M5-T10, M4-T4).
• Dynamic objects. Features on moving people or vehicles. If they form the largest rigid group, RANSAC may select them as the model; the robot appears to move opposite to the vehicle. Mitigate with semantic masks, IMU consistency and robust kernels.
• Reflections and transparent surfaces. Features on reflections move differently from the scene and appear at the wrong depth.

Geometric failures

• Pure rotation. No baseline: depth cannot be triangulated and the map points stop being created; tracking continues for a while on the existing points and then fails when they leave the view (M6-T3).
• Planar or near-planar scenes. The essential matrix is degenerate; use the homography.
• Small parallax or far scenes. Depth is ill-conditioned; all points are effectively at infinity and only rotation is observed.
• Forward motion along the optical axis (driving straight): the image expands from the focus of expansion with little lateral parallax; depth near the axis is poorly constrained.
• Collinear or clustered features. The pose has weak directions.

Sensor and estimator failures

• Rolling shutter during fast motion: skewed geometry and larger residuals (M6-T8).
• Calibration and time offsets: biases that depend on the motion (M6-T8).
• IMU saturation or bad initialisation: the scale or gravity is wrong from the start, and the system never recovers until a reset.
• Scale drift in monocular systems: the trajectory shrinks or stretches over time and is only corrected by a loop closure with Sim(3) (M6-T5).
• Map contamination: a wrong pose adds wrong map points, which then attract further wrong matches.

Relocalisation

After tracking is lost the system looks up the current image in the bag-of-words database and verifies a candidate with PnP and RANSAC (M6-T5). Failure modes: the place was never mapped or looks different (lighting, season, moved furniture); the candidate is aliased; the camera has moved far and the first view has too few matches. A false relocalisation is worse than none, because the system tracks confidently in the wrong place: always require a minimum inlier count and consistency over several frames.

inlier count

number of features consistent with the estimated pose; the primary health signal.

parallax

angular or pixel shift of features between keyframes; low parallax means ill-conditioned depth.

f·ω·t

motion blur length in pixels for rotation rate ω and exposure t.

FOE

focus of expansion: the image point toward which the camera moves.

Signals and mitigations

A worked example: reading a failed run

1. Blur. A handheld camera with f = 600 px pans at ω = 1 rad/s with a 20 ms exposure. Blur length = 600 × 1 × 0.020 = 12 px. With a 5 ms exposure it is 3 px. The first is blur across a 12 px smear (corners vanish), the second is tolerable. The cost of the shorter exposure is more noise in dim light: a trade-off.
2. Reading logs. A run reports tracked features per frame: 220, 215, 218, 190, 120, 60, 22, then LOST. Parallax and residual stay normal until frame 4. The decline is steady over six frames and correlates with the gyroscope rate rising from 0.3 to 1.8 rad/s: blur from a fast turn, not a sudden occlusion. A sudden drop from 220 to 20 in one frame would instead suggest an exposure jump or a lens cover.
3. Dynamic objects. A robot waits at a junction while a bus passes. The bus fills 60% of the image with 130 features and the static background has 90. RANSAC finds the largest consistent set, which is the bus. The estimated camera velocity opposes the bus’s motion, while the IMU says the robot is stationary. A check that the visual rotation agrees with the gyroscope within a threshold would reject this model.
4. Pure rotation. The robot spins in place. After 40° of rotation there are no new map points (parallax is zero), and the old points leave the field of view. Tracked inliers fall to fewer than 15 and the system is lost although the images are sharp. The fix is design, not tuning: rotate slowly while translating, or add an IMU and accept that scale is not updated during the spin.
5. False recovery. After a loss, relocalisation matches the image to a keyframe of an identical aisle with 38 inliers. The threshold was 30. The system resumes tracking 12 m away from the truth and corrupts the map. A rule such as “at least 50 inliers and three consecutive consistent candidates” would have rejected it.

Engineering implications and limitations

Engineering implications

• Instrument the system. Log feature counts, inliers, parallax, exposure, state transitions, bias estimates and covariance for every run. Failures that cannot be seen in logs cannot be fixed.
• Define health thresholds on those signals and act before the system is lost: slow down, stop, switch to another source (wheel odometry, lidar), or stop adding keyframes.
• Control the camera. Fix or bound exposure, use a short exposure and a high frame rate, prefer a global shutter for fast platforms.
• Keep an independent fallback. An IMU, wheel odometry or lidar to bridge outages and to cross-check the visual pose.
• Do not accept recoveries blindly. Require high inlier counts and temporal consistency; log all relocalisation events.
• Test in the failure conditions (dim light, glare, moving crowds, long corridors) before deployment, not only in the lab.

Limits

No mitigation removes the dependence on the visual front end: if the camera sees nothing distinctive, a camera-only system has nothing to work with. Redundant sensing is the dependable answer. And the failure analysis here assumes feature-based systems; direct and learned systems fail differently, for example with photometric changes that features survive, or with out-of-distribution scenes for a network.

Common misconceptions

Engineering takeaways

1. Visual SLAM failures are chains from image quality to features to matches to geometry to estimation; look for the first broken link.
2. Blur is about f·ω·t pixels; low texture, aliasing, dynamic objects, pure rotation and rolling shutter are the common causes.
3. The logs (inliers, parallax, residuals, bias estimates) degrade before the state flag changes.
4. Relocalisation needs verification; a false recovery is dangerous.
5. Redundant sensing and instrumentation are the practical defences.

References and further reading

Primary papers and authoritative documentation.

• Cadena, C. et al. (2016). Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics 32(6).
• Mur-Artal, R., Montiel, J. M. M. and Tardós, J. D. (2015). ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics 31(5).
• Scaramuzza, D. and Fraundorfer, F. (2011). Visual odometry: Part I: The first 30 years and fundamentals. IEEE Robotics & Automation Magazine 18(4).
• Fraundorfer, F. and Scaramuzza, D. (2012). Visual odometry: Part II: Matching, robustness, optimization, and applications. IEEE Robotics & Automation Magazine 19(2).
• Saputra, M. R. U., Markham, A. and Trigoni, N. (2018). Visual SLAM and structure from motion in dynamic environments: A survey. ACM Computing Surveys 51(2).
• Engel, J., Usenko, V. and Cremers, D. (2016). A photometrically calibrated benchmark for monocular visual odometry. arXiv:1607.02555.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes the theory of visual and visual-inertial SLAM. The four labs ask you to diagnose feature tracking, quantify visual odometry drift, read the logs of a failed tracker and estimate a camera–IMU time offset.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T8: Sensor Calibration and Synchronization

  Describe intrinsic, extrinsic and temporal calibration and the errors each produces when wrong.

• Lab
  M6-L3: Investigate ORB-SLAM Failure Modes

  Given the state logs of a keyframe-based tracker, identify why tracking was lost and how it relocalised.