M8-T6 — Learning-Based SLAM

Learning objectives

After this lesson, you should be able to:

• List the SLAM components that learning has improved and name representative methods for each.
• Distinguish learned components inside a geometric pipeline from end-to-end and differentiable SLAM.
• Explain why generalisation and failure prediction are the central risks of learned SLAM.
• Read a learned-SLAM paper critically: data, benchmarks, compute and ablations.

The problem and the intuition

A hand-designed corner detector (M6-T1) is a rule: “the structure tensor has two large eigenvalues”. It was designed for generic images and works in none of them perfectly. A network trained on millions of image pairs can learn what is repeatable in the kinds of scenes it has seen, including under changes of lighting that break the rule. The same holds for matching (learn which pairs correspond), for place recognition (learn descriptors invariant to season and weather), and for depth (learn typical sizes and shapes).

The trade is between performance on the distribution and guarantees off it. A geometric method fails in understood ways: no texture, pure rotation, degeneracy. A learned method fails in ways nobody has listed: a new kind of building, an unusual camera, a strong rain pattern. The design principle that follows is to learn the parts that are perceptual (what is the same point?) and keep the parts that are geometric (what motion explains these points?), or, if both are learned, to keep a geometric check on the output.

Learn perception, verify with geometry. A learned component inside a geometric pipeline is routine; replacing the pipeline is research.

What is learned, and how well it is established

Problem. Improve the robustness and accuracy of SLAM components, or of the whole system, with models trained on data, without losing the reliability that geometry provides.

Assumptions. Training data resemble the deployment conditions (cameras, lighting, scenes); compute (often a GPU) is available at runtime; the learned output is used with uncertainty or verified geometrically.

Established as components

These components feed a conventional geometric back end: the learned matcher produces correspondences, RANSAC and bundle adjustment (M4-T7, M6-T3) estimate the motion. The outputs are verified by geometry, so a wrong match is caught by the same mechanisms that catch any outlier.

Research: learned odometry and differentiable SLAM

• End-to-end odometry. DeepVO (Wang et al., 2017) regresses pose from image pairs with a recurrent network. It was an early demonstration and is surpassed by geometric methods on standard benchmarks, mainly generalising poorly.
• Learned structure with geometric optimisation. CodeSLAM (Bloesch et al., 2018) learns a compact depth code optimised jointly with pose. BA-Net (Tang and Tan, 2019) makes bundle adjustment differentiable inside a network. D3VO (Yang et al., 2020) adds predicted depth, pose and uncertainty to a direct VO system.
• DROID-SLAM (Teed and Deng, 2021) iteratively updates dense optical flow and depth with a recurrent unit and a differentiable bundle-adjustment layer, achieving strong results across several benchmarks with a single trained model for monocular, stereo and RGB-D input. DPVO (Teed et al., 2023) uses sparse patches to cut the cost. They are research systems with notable generalisation, which still require a GPU and give limited insight into failures.

min over (T, X) Σ ρ( ‖ π(T X) − u ‖² )

Reading the evidence

• Data. Is the test set from the same distribution as training? Is the camera the same?
• Benchmark and protocol. Aligned how (M8-T8)? Which sequences were left out? Are failures included?
• Compute. GPU memory, latency and power; a learned system that needs 10 GB of GPU memory is not the same product as a CPU system.
• Ablations. Which learned part gives the gain, and what happens with a classical replacement?
• Failure behaviour. Does the paper show failure cases, or a way to detect them?

SuperPoint

a network that detects keypoints and computes descriptors in one pass.

attention matcher

a network (SuperGlue, LightGlue) that matches two sets of keypoints using context from all of them.

differentiable BA

a bundle-adjustment layer through which gradients flow so the network can be trained to help it.

domain shift

the difference between training and deployment data that degrades learned models.

A worked example: cost of attention matching and a sanity check

1. Matching cost. A classical nearest-neighbour search between two sets of 2 000 binary descriptors needs 2 000 × 2 000 = 4 × 10⁶ Hamming distances; each is an XOR and a population count (M6-T1), a few milliseconds on a CPU. An attention matcher computes pairwise interactions among all 4 000 keypoints in each of several layers: tens of millions of multiply-adds per layer with floating-point descriptors of 256 values, which is why it needs a GPU for real time. LightGlue reduces this by exiting early and pruning keypoints.
2. Why geometry still verifies. Suppose a learned matcher gives 400 matches of which 5% are wrong (20 wrong). RANSAC with the 5-point solver and w = 0.95 needs only N = ln 0.01 / ln(1 − 0.95⁵) = 4.6 / 1.49 ≈ 4 iterations (M6-T2), against 145 at 50% inliers. A better matcher cuts the geometric cost as well as the errors.
3. Domain shift in numbers. A place-recognition descriptor reaches 95% recall at 100% precision on daytime streets. In the same city at night, recall drops to 60% (an illustrative number, not a measurement from a paper). If the system sets its acceptance threshold on daytime data, it will accept no loop closures at night; if it lowers the threshold, it will accept wrong ones. The right fix is to calibrate on night data or to combine the descriptor with a geometric verification.
4. A learned depth prior. A network predicts depth for a monocular camera with a median relative error of 10% on its training domain. Used to initialise a monocular system, it fixes the scale to within 10%. Out of domain, the median error may reach 30%, still better than no prior but not enough for metric claims; combine with an IMU (M6-T6).
5. Uncertainty. DROID-style systems predict per-pixel confidence weights W. If the weights are uncalibrated, the optimiser trusts bad regions. Compare the predicted confidence with the actual error on held-out data, as one would check NEES for a filter (M8-T8).

Engineering implications and limitations

Engineering implications

• Adopt learned components where they are proven: keypoints and matchers for difficult illumination, place recognition across seasons, depth priors for initialisation.
• Keep geometric verification. RANSAC, reprojection checks and consistency tests catch the errors that a network makes with confidence.
• Measure on your own data. Test with your camera, your lighting and your environments, including the worst conditions. Published numbers do not transfer automatically.
• Budget the compute. A GPU, its power and its thermal limits are part of the system (M7-T8).
• Track the model version and the training data; retraining changes behaviour and must be re-evaluated like a new SLAM algorithm.
• Plan for failure. Learned systems give no warning of failures that classical ones signal (low inlier counts, degenerate eigenvalues); add independent monitors.

Limits

Learned components lack guarantees: there is no theory that predicts when a network will fail, and uncertainty estimates are often miscalibrated. End-to-end systems have shown strong benchmark results but depend on large GPUs and on training distributions, and their long-term and large-scale behaviour is less studied. Fair comparisons are hard (different training data, different tuning budgets), and results change quickly, so treat any table of rankings as a snapshot and look for independent re-evaluations.

Common misconceptions

Engineering takeaways

1. Learned keypoints, matchers, place-recognition descriptors and depth priors are established components inside geometric pipelines.
2. End-to-end and differentiable SLAM (DeepVO, CodeSLAM, BA-Net, DROID-SLAM, DPVO) is active research with strong benchmarks but heavy compute and limited failure insight.
3. Learn perception, verify with geometry; a better matcher also cuts RANSAC cost.
4. Domain shift is the central risk; evaluate on your data and monitor independently.
5. Read learned-SLAM papers for data, protocol, compute, ablations and failure cases.

References and further reading

Primary papers. In the text, learned components inside geometric pipelines (established) are separated from end-to-end and differentiable systems (active research).

• DeTone, D., Malisiewicz, T. and Rabinovich, A. (2018). SuperPoint: Self-supervised interest point detection and description. CVPR Workshops.
• Sarlin, P.-E., DeTone, D., Malisiewicz, T. and Rabinovich, A. (2020). SuperGlue: Learning feature matching with graph neural networks. IEEE CVPR.
• Lindenberger, P., Sarlin, P.-E. and Pollefeys, M. (2023). LightGlue: Local feature matching at light speed. IEEE ICCV.
• Arandjelović, R., Gronat, P., Torii, A., Pajdla, T. and Sivic, J. (2016). NetVLAD: CNN architecture for weakly supervised place recognition. IEEE CVPR.
• Teed, Z. and Deng, J. (2021). DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D cameras. Advances in Neural Information Processing Systems 34.
• Teed, Z., Lipson, L. and Deng, J. (2023). Deep patch visual odometry. Advances in Neural Information Processing Systems 36.
• Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S. and Davison, A. J. (2018). CodeSLAM: Learning a compact, optimisable representation for dense visual SLAM. IEEE CVPR.
• Wang, S., Clark, R., Wen, H. and Trigoni, N. (2017). DeepVO: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. IEEE ICRA.
• Godard, C., Mac Aodha, O., Firman, M. and Brostow, G. (2019). Digging into self-supervised monocular depth estimation. IEEE ICCV.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A different use of learning concerns the map itself. The next lesson looks at neural and implicit map representations, from radiance fields to Gaussian splatting, and what they change for SLAM.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T5: GPS-Denied Localization

  Combine map matching, inertial and visual odometry and radio/magnetic aids when GNSS is lost.

• Lab
  M8-L4: Evaluate a SLAM System Under Controlled Failures

  Inject controlled failures (noise, outliers, dropouts) into a SLAM run and report degradation honestly.