M8-T12: Visual odometry failure handling

Learning objectives

After this lesson, you should be able to:

• List the causes of VO failure and the signature each leaves in observable quantities.
• Design a multi-indicator health monitor with thresholds and hysteresis.
• Compute the time available in inertial fallback from the drift budget.
• Verify the monitor with fault injection and report detection delay and false alarms.

Conceptual explanation

Implementation-specific Indicators and thresholds are system-specific: the values in the example are placeholders to be tuned and validated for the actual camera, VIO algorithm and environment. VIO systems expose different diagnostics; check what yours provides.

Typical failure causes and how they appear:

The most informative single check is consistency with the inertial prediction: the VIO velocity or the visual position increment should agree with what the IMU predicts within the expected uncertainty. A normalised innovation (as in the EKF lesson) that stays high flags a failure that count-based indicators miss, such as wrong but consistent matches.

Engineering intuition

No single indicator is reliable: a high feature count on a repetitive pattern is dangerous, and a low count on a clean, well-conditioned scene may be fine. Combine indicators so that a failure shows up in at least two, and make the decision conservative in the direction of safety: enter degraded mode quickly, leave slowly (hysteresis).

The fallback time is an arithmetic fact: after vision is lost, the position error grows with the inertial drift. A velocity error $v_{e}$ grows position error linearly, and a constant accelerometer bias $b$ grows it as ½ $b\,t^{2}$. The bounded fallback is whichever limit the mission’s accuracy or clearance allows; when it is reached, the vehicle must already be doing something safe (hold in a known-safe place, land, or return on a path that is safe without precise position).

Mathematical formulation

Inertial dead-reckoning error after vision loss at time 0, with initial velocity error $v_{0}$, accelerometer bias $b$ (after calibration), attitude error $\theta _{0}$ (which leaks gravity into horizontal acceleration as $g\,\theta$) and velocity random walk $\sigma _{v}$:

The time to reach a drift budget $D$ is found by solving $e_{p}(t)=D$. With $v_{0}=0.05\,\mathrm{m}/\mathrm{s}$, $b+g\,\theta _{0}=0.05\,\mathrm{m}/\mathrm{s}^{2}$ (a modest $0.3^{\circ}$ tilt error and a small bias) and $D=2\,\mathrm{m}$, $t\approx 8.3\,\mathrm{s}$ ignoring noise: the vehicle has seconds, not minutes, which is why the safe action must start immediately. Detection delay counts against the same budget: with indicator confidence updated at frame rate $f$ and requiring $m$ consecutive bad frames, the delay is m/f; a combined delay of 0.3 s costs about ½·0.05·0.09 + 0.05·0.3 ≈ 0.017 m, small compared with the budget. The false-alarm rate is $p_{\text{fa}}^{m}$ per window if bad frames are independent, which motivates $m\ge 3$.

$v_{0}$, $\theta _{0}$

initial velocity and attitude errors at the moment of loss.

$b$

residual accelerometer bias.

$\sigma _{v}$

velocity random walk of the accelerometer $(m/s/\sqrt{s})$.

$D$

allowed position drift budget.

Architecture and data flow

VIO front end diagnostics (tracked features, inliers, reprojection error) + image statistics (sharpness, mean intensity, saturated fraction, frame timestamps) + EKF innovations for the vision updates → health monitor → confidence level and reason code → flight controller (stop fusing external vision, change mode) and mission layer (slow down, hold, land). The decision, its reason and the indicators are logged at every change for post-flight analysis, and the dropout is reported to the operator.

Real-world UAV applications

• Handling featureless corridors and lighting changes in GNSS-denied flights.
• Deciding when to stop fusing visual odometry into the autopilot.
• Setting the time limit for inertial fallback in the mission plan.
• Fault-injection tests of the vision pipeline.

Implementation example

This runs as shown. A multi-indicator health monitor on a synthetic sequence with three injected faults: a featureless period, a repetitive-texture period (features present, but matches inconsistent with the IMU) and a blur burst. It reports detection delay per fault and false alarms in fault-free periods, and the fallback time budget from the drift formula.

import numpy as np
rng = np.random.default_rng(41)
fps, T = 20, 120; n = fps * T; t = np.arange(n) / fps
feat = 150 + 15 * rng.standard_normal(n); inl = 0.9 + 0.03 * rng.standard_normal(n); sharp = 1.0 + 0.05 * rng.standard_normal(n); nis = rng.chisquare(3, n) / 3     # fault-free indicators (nis: normalised innovation squared, mean 1)
def seg(a, b): return (t >= a) & (t < b)
f1, f2, f3 = seg(30, 38), seg(60, 70), seg(95, 99)
feat[f1] = 12 + 3 * rng.standard_normal(f1.sum())                                   # fault 1: featureless wall: few features
inl[f2] = 0.45 + 0.05 * rng.standard_normal(f2.sum()); nis[f2] = 9 + 2 * rng.standard_normal(f2.sum())      # fault 2: repetitive texture: features fine, inliers and consistency bad
sharp[f3] = 0.35 + 0.05 * rng.standard_normal(f3.sum()); feat[f3] = 40 + 8 * rng.standard_normal(f3.sum())  # fault 3: blur burst
def bad(i):
    flags = [feat[i] < 60, inl[i] < 0.7, sharp[i] < 0.6, nis[i] > 5.0]
    return sum(flags) >= 1 and (sum(flags) >= 2 or feat[i] < 30 or nis[i] > 5.0)          # at least one strong indicator, or two weak ones
state = 'good'; bad_run = 0; good_run = 0; events = []
for i in range(n):
    if bad(i): bad_run += 1; good_run = 0
    else: good_run += 1; bad_run = 0
    if state == 'good' and bad_run >= 3: state = 'lost'; events.append((t[i], 'lost'))
    elif state == 'lost' and good_run >= 20: state = 'good'; events.append((t[i], 'good'))   # leave slowly: 1 s of clean frames
for name, f in (('featureless', f1), ('repetitive', f2), ('blur', f3)):
    start = t[f][0]; det = next((te for te, s in events if s == 'lost' and te >= start), None)
    print(f"{name:>11}: starts at {start:5.1f} s, declared lost at {det:5.2f} s (delay {det - start:.2f} s)")
fa = [te for te, s in events if s == 'lost' and not (f1 | f2 | f3)[min(int(te * fps), n - 1)] and not (f1 | f2 | f3)[max(int(te * fps) - 5, 0)]]
print("false alarms in fault-free periods:", len(fa))
assert len(fa) == 0 and all(next(te for te, s in events if s == 'lost' and te >= t[f][0]) - t[f][0] < 0.5 for f in (f1, f2, f3))
v0, bg, D = 0.05, 0.05, 2.0; tt = np.sqrt(2 * D / bg) if v0 == 0 else (-v0 + np.sqrt(v0 ** 2 + 2 * bg * D)) / bg
print(f"inertial fallback time to reach {D} m of drift (v0 = {v0} m/s, bias-equivalent {bg} m/s^2): {tt:.1f} s")
assert 6 < tt < 9

Common engineering mistakes

Failure modes

Debugging approach

1. Plot every indicator with the ground-truth failure intervals and the monitor’s decision.
2. Compute detection delay and false-alarm rate on recorded data.
3. Check innovation consistency against the IMU independently of the VIO’s own confidence.
4. Replay failure recordings through the monitor offline after each change.

Practical exercises

1. Tune the thresholds on a recording of your own VIO and report detection delay and false alarms.
2. Compute the fallback time for $D=1\,\mathrm{m}$ and a 0.2 m/s initial velocity error.
3. Add an exposure-saturation indicator and a test with a flashlight.
4. Write the mode-transition table for the VO confidence levels with the autopilot’s modes.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• Forster, C., Carlone, L., Dellaert, F. & Scaramuzza, D. (2017). On-manifold preintegration for real-time visual-inertial odometry. IEEE T-RO 33(1)
• Mur-Artal, R. & Tardós, J. D. (2017). ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE T-RO 33(5)
• PX4: Visual inertial odometry (VIO)

What’s next