M8-T2 — Semantic SLAM

Learning objectives

After this lesson, you should be able to:

• Describe three roles of semantic information in SLAM: data association, object-level mapping and masking.
• Compute a simple class-aware association probability and explain why it improves on geometry alone.
• Compare dense-semantic, object-level and factor-graph approaches and cite the representative systems.
• State the risks of semantic SLAM, in particular detector errors, and which parts are still research.

The problem and the intuition

Imagine returning to a room and trying to decide whether the chair you see is the one you saw before. If you only compare positions, a chair three centimetres from where you remember it is “the same”, but so would a small table at the same spot. If you also compare what it is, the question becomes much easier. Data association (M2-T7) is the weakest link in SLAM: one wrong association can corrupt the map. Class labels give an independent piece of evidence, one that does not depend on viewpoint the way appearance descriptors do.

Semantics can also change the unit of the map. Instead of 10⁶ points, the map holds fifty objects with pose and shape: smaller, easier to reason with, easier to update when one object moves. The price is that the map now depends on a recogniser that can be wrong, and a wrong label can be as damaging as a wrong match.

Semantics add independent evidence for association, a compact object-level map, and a way to mask movable things. They also add a new source of error: the recogniser.

Roles, formulations and systems

Problem. Use object or class information produced by a recogniser to improve localisation and mapping, and to build maps that represent entities rather than only geometry.

Assumptions. A detector or segmentation network gives classes (and possibly instance masks and boxes) at useful accuracy for the environment; objects have a persistent identity that the system can track; for object-level landmarks, an object shape model (a CAD library, an ellipsoid) is available or estimated.

Role 1: data association

Given a detection d of class c_d at a measured pose, and map objects m_j of class c_j, a probabilistic association assigns weights:

where the first factor is a confusion model (the detector’s probability of reporting class c_d when the true class is c_j) and the second is the geometric likelihood (M2-T7, Mahalanobis distance). Bowman et al. (2017) treat the discrete association as latent and estimate it with expectation–maximisation together with the poses, so ambiguous associations are weighted, not hard-decided. This is established in the research literature and is beginning to appear in systems; it is not yet a default feature of production SLAM.

Role 2: object-level landmarks

• SLAM++ (Salas-Moreno et al., 2013): a database of known 3D object models; objects detected in an RGB-D stream become nodes in a pose graph, with relative-pose edges to the camera. Works for known objects; compact; real time.
• QuadricSLAM (Nicholson et al., 2019): landmarks are dual quadrics (ellipsoids) estimated from 2D object detections in several views, inside a factor graph, so no CAD models are needed.
• Fusion++ (McCormac et al., 2018): instance-level volumetric object maps built online from RGB-D with learned instance segmentation.

Role 3: dense semantic maps and masks

LiDAR systems such as SuMa++ (Chen et al., 2019) segment each scan semantically with a network and use the labels to remove moving classes and to weight the registration residuals by class (road and building points constrain the pose; vegetation is noisy). Kimera (Rosinol et al., 2020) builds a metric-semantic mesh on top of visual-inertial odometry. Visual systems mask classes such as people (M8-T1).

P(c_d | c_j)

confusion probability: how often the detector says c_d for a true class c_j.

h(m_j)

measurement predicted for map object m_j from the current pose.

quadric

a quadric surface, here an ellipsoid summarising an object’s extent.

instance

one particular object (this chair), as opposed to a class (chairs).

Established practice and active research

Failure modes

• Detector errors become estimator errors. A false positive object adds a spurious landmark; a missed object adds none; a wrong class breaks the association it was supposed to protect.
• Domain shift. A network trained on street scenes may fail indoors; failures are not announced, and confidence scores are poorly calibrated.
• Identity over time. Instances must be tracked across views; two chairs of the same type are the classic confusion.
• Computation. Running a network per frame costs GPU time and latency (M7-T8).

A worked example: class evidence resolves a geometric tie

1. The scene. The map holds a chair at (2.00, 1.00) and a table at (2.30, 1.00). The robot detects an object at (2.15, 1.00) of class “chair”, with geometric uncertainty σ = 0.20 m in x.
2. Geometry alone. The detection is 0.15 m from the chair and 0.15 m from the table. The Gaussian likelihood is the same for both: exp(−0.5 × (0.15/0.20)²) = exp(−0.28) = 0.755. Weights: 0.5 and 0.5. The association is a coin toss, and a hard decision would be wrong half of the time.
3. With class evidence. Suppose the detector reports the true class correctly 85% of the time and confuses chair and table 10% of the time. P(“chair” | chair) = 0.85 and P(“chair” | table) = 0.10. The weights become 0.85 × 0.755 = 0.642 for the chair and 0.10 × 0.755 = 0.0755 for the table. Normalised: 0.895 and 0.105. The chair is now clearly preferred, with 10.5% remaining for the table: still honest about the ambiguity.
4. When the detector is wrong. If it mislabels the table as a chair (probability 0.10 per detection of the table), the system tends to pull the table landmark toward the wrong object. Using the soft weights with a robust loss (M4-T5) limits the damage to a small pull; a hard assignment would add a wrong edge at full strength.
5. Map size. A room scanned at 5 mm resolution has about 10⁷ points; as 40 object landmarks each with a 9-parameter ellipsoid (centre 3, axes 3, orientation 3), the object map has 360 numbers. The object map is about 10⁵ times smaller, but it says nothing about the floor or the walls; the two representations are complementary.

Engineering implications and limitations

Engineering implications

• Start with masking. It is the cheapest, most robust use of semantics: drop features and points on classes that move.
• Use soft evidence. Combine class probabilities with geometric likelihood and robust losses; avoid hard gating on a label unless the detector is validated in your environment.
• Measure the detector on your data. Build a confusion matrix for your classes and lighting; use it as the P(c_d | c_j) in the association.
• Keep geometry as the backbone. Semantic constraints should supplement, not replace, the geometric ones, so that a detector failure degrades the system gracefully.
• Budget the compute. Run the detector on keyframes or at a lower rate where possible, and on an accelerator.
• Log detections with confidence so that failures can be reviewed (M8-L4).

Limits

Semantic SLAM inherits the limitations of its recogniser: closed vocabularies, domain shift and unreliable confidence. Evidence that semantics improve accuracy is strongest in dynamic or cluttered scenes and weaker in benign ones, where geometry alone is already good. Claims of robustness should be tested with the detector degraded (M8-L4). The theoretical treatment of semantic landmarks, in particular observability of object poses from few views and consistency of object-level estimators, is incomplete.

Common misconceptions

Engineering takeaways

1. Semantics help SLAM in three ways: class-aware data association, object-level landmarks, and masking of movable classes.
2. Class evidence turns a geometric tie into a clear preference, but remains probabilistic: combine it with the geometric likelihood and robust losses.
3. Established: masking and class-weighted registration. Active research: probabilistic association, generic object SLAM, open-vocabulary semantics.
4. Detector errors become estimator errors; validate the detector in your environment and degrade gracefully.
5. Keep geometry as the backbone; semantic constraints supplement it.

References and further reading

Primary papers. The table in the text marks which uses of semantics are established and which are research.

• Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. J. and Davison, A. J. (2013). SLAM++: Simultaneous localisation and mapping at the level of objects. IEEE CVPR.
• Bowman, S. L., Atanasov, N., Daniilidis, K. and Pappas, G. J. (2017). Probabilistic data association for semantic SLAM. IEEE ICRA.
• Nicholson, L., Milford, M. and Sünderhauf, N. (2019). QuadricSLAM: Dual quadrics from object detections as landmarks in object-oriented SLAM. IEEE Robotics and Automation Letters 4(1).
• McCormac, J., Clark, R., Bloesch, M., Davison, A. and Leutenegger, S. (2018). Fusion++: Volumetric object-level SLAM. International Conference on 3D Vision (3DV).
• Chen, X. et al. (2019). SuMa++: Efficient LiDAR-based semantic SLAM. IEEE/RSJ IROS.
• Rosinol, A., Abate, M., Chang, Y. and Carlone, L. (2020). Kimera: An open-source library for real-time metric-semantic localization and mapping. IEEE ICRA.
• Cadena, C. et al. (2016). Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics 32(6).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The robot is not alone in many applications. The next lesson merges maps across sessions and robots, where the central problem is deciding which of the claimed matches between maps to believe.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T1: Dynamic Environments

  Handle moving objects: detection, filtering, and dynamic-aware mapping.

• Lab
  M8-L1: Handle Dynamic Objects in Mapping

  Moving people leave ghost obstacles in the map: remove them with a consistency check.