Skip to the lesson
FixTheRobot Explore Robotics / Localization & SLAM / M6 Visual and Visual-Inertial SLAM / M6-T5

Localization & SLAM · Theory · M6

M6-T5 — ORB-SLAM Architecture

A real-time visual SLAM system is mostly bookkeeping: which frames to keep, which points to trust, and which work can wait.

  • Lesson 5 of 9
  • Module 6: Visual and Visual-Inertial SLAM
  • Theory
  • About 10 minutes to read

ORB-SLAM (Mur-Artal, Montiel and Tardós, 2015) is the reference feature-based visual SLAM system. Its power is not one clever algorithm but an architecture: three cooperating threads around a keyframe map, a graph that records which keyframes see the same points, and strict rules for creating and deleting what it stores. This lesson walks through the design and then through what goes wrong.

Learning objectives

After this lesson, you should be able to:

  • Describe the roles of the tracking, local mapping and loop closing threads and what each does at what rate.
  • Explain keyframes, map points, the covisibility graph and the essential graph.
  • State the rules for keyframe insertion and for culling map points and keyframes.
  • Outline place recognition, Sim(3) loop correction and relocalisation, and the extensions in ORB-SLAM2 and ORB-SLAM3.

The problem and the intuition

A camera produces 30 frames a second, but a robot walking down a corridor does not learn 30 new things a second: consecutive frames see nearly the same scene. Optimising over all of them would waste effort. The architectural idea of keyframe SLAM is to keep only the frames that add information (the keyframes), maintain the map from those, and let each new frame be tracked against the map rather than against the previous frame.

That split also separates the time scales. Tracking must finish before the next image arrives. Adding map points and optimising a local window can take a few frames. Detecting a loop and correcting the whole map may take a second or more. Running them in separate threads lets the pose be output at frame rate while the map improves in the background.

Track every frame, map every keyframe, close loops when you can: three tasks at three speeds around one shared map.

ORB-SLAM threads and shared mapThree threads share a map of keyframes and map points. Tracking estimates the pose of each frame and decides on keyframes. Local mapping creates points and runs local bundle adjustment. Loop closing detects revisits with a bag of words and corrects the map.Trackingevery frame: ORB, pose, keyframe? Local mappingnew points, local BA, culling Loop closingDBoW2, Sim(3), essential graph Shared mapkeyframes · map points · covisibility graph · spanning tree · essential graph · bag-of-words database Tracking runs at frame rate; the other two threads work on keyframes and may lag.Relocalisation (after lost tracking) reuses the bag-of-words database and PnP.
ORB-SLAM: three threads and one shared map.

The architecture and its rules

Problem. Estimate the camera trajectory and a sparse 3D map in real time from monocular, stereo or RGB-D input, in large environments, with loop closure and recovery from lost tracking.

Assumptions. A calibrated camera; a static, textured scene; enough overlap between consecutive frames; a bag-of-words vocabulary trained offline on a large set of images of similar scenes.

The map

  • Map points: 3D positions with a representative ORB descriptor and the keyframes that observe them.
  • Keyframes: a camera pose, the intrinsics, and all ORB features extracted in that frame.
  • Covisibility graph: keyframes are nodes; an edge links two keyframes that share at least 15 map points, weighted by the number shared. It defines the local neighbourhood used for tracking and local optimisation.
  • Essential graph: a sparse subgraph made of the spanning tree, strong covisibility edges (at least 100 shared points) and loop edges. Loop correction optimises this graph rather than the full one.

Tracking thread (every frame)

  1. Extract ORB features (several scales, evenly spread over the image).
  2. Predict the pose with a constant-velocity model, project the map points seen in the previous frame and search for matches in a small window; or, if tracking was lost, relocalise (below).
  3. Optimise the pose alone (motion-only bundle adjustment), minimising reprojection error with a robust cost.
  4. Project the local map (points of the reference keyframe and its covisible neighbours) into the frame, find more matches, and optimise again.
  5. Decide whether this frame should become a keyframe.

Keyframe insertion rule (ORB-SLAM, 2015): all of the following hold: more than 20 frames since the last global relocalisation; local mapping is idle or more than 20 frames have passed since the last keyframe; the current frame tracks at least 50 points; and it tracks fewer than 90% of the points of the reference keyframe, meaning that the view has changed enough to be worth adding. Insertion is generous and culling is strict.

Local mapping thread (every keyframe)

  1. Insert the keyframe into the covisibility graph and update the spanning tree.
  2. Cull recent map points. A new point survives only if it is found in more than 25% of the frames where it is predicted visible, and it is observed from at least three keyframes within the first frames after its creation. This removes points created from bad matches.
  3. Create new points by triangulating unmatched features between this keyframe and its covisible neighbours, checking parallax, positive depth and reprojection error.
  4. Local bundle adjustment over the current keyframe, its covisible keyframes and all points they see; other keyframes that observe those points stay fixed.
  5. Cull keyframes. Delete a keyframe if 90% of its points are seen by at least three other keyframes at a similar or finer scale: it is redundant.

Loop closing thread

  1. Detect. Convert the new keyframe’s descriptors into a bag-of-words vector (DBoW2, Gálvez-López and Tardós, 2012), query the database for similar keyframes that are not covisible, and accept a candidate only if it is consistent with two further keyframes (temporal consistency).
  2. Compute the Sim(3). For a monocular system, estimate a similarity transform (rotation, translation, scale) from the 3D-to-3D matches with RANSAC, then refine. For stereo and RGB-D, a rigid SE(3) suffices because the scale is observed.
  3. Fuse and correct. Merge duplicated points, add edges to the covisibility graph, propagate the correction through neighbours, and optimise the essential graph (a pose-graph optimisation, Module 4). Finally run a full bundle adjustment.

Relocalisation

When tracking is lost (too few matches), the system queries the bag-of-words database with the current frame, takes candidate keyframes, and solves PnP with RANSAC (M6-T3) from the 2D-to-3D matches. If enough inliers are found, tracking resumes. This depends on the stored descriptors and on the place being recognisable from the new view.

Extensions

ORB-SLAM2 (2017) adds stereo and RGB-D input with close and far points (M6-T4). ORB-SLAM3 (Campos et al., 2021) adds tightly coupled visual-inertial estimation (M6-T6, M6-T7), fisheye lenses, and the Atlas: when tracking is lost and cannot be recovered, a new map is started instead of failing, and maps are merged when they are later found to overlap.

motion-only BA
(R, t)∗=arg min⁡ ∑i ρ(∥ui−π(K(R Xi+t)) ∥Σ2)(R,\,t)*=\operatorname*{arg\,min}\,\sum _{i}\,\rho (\lVert u_{i}-\pi (K(R\,X_{i}+t))\,\rVert ^{2}_{\Sigma })
over the matched map points Xi\text{over the matched map points}\,X_{i}
covisibility graph
keyframes linked when they observe common map points; defines local neighbourhoods.
essential graph
sparse subgraph used for fast loop correction.
Sim(3)
similarity transform (rotation, translation and scale), the drift group of monocular SLAM.
DBoW2
bag-of-binary-words library for place recognition with ORB descriptors.

Strengths, limitations, failure modes

StrengthLimitation / failure mode
Real-time on a CPU; robust to wide baselines through feature descriptorsFails in textureless scenes, under heavy blur and rapid illumination change
Full lifecycle: initialise, track, map, close loops, relocalise, merge mapsInitialisation (monocular) needs parallax and a non-degenerate scene; slow motion at the start causes failure
Sparse map is compact and good for localisationSparse map gives no obstacle information; dense reconstruction needs another module
Strict culling keeps the map cleanDynamic objects can enter the map; perceptual aliasing can give a false loop that the Sim(3) check does not reject

A worked example: insertion rule, culling and the covisibility graph

  1. Insertion. The reference keyframe has 300 map points. The current frame tracks 250 of them: 250 / 300 = 83% < 90%, and 250 ≥ 50. Assuming the timing conditions hold, a new keyframe is inserted. If the frame tracked 290 points (96.7%), nothing new would be seen and no keyframe would be inserted.
  2. Point culling. A new point is predicted to be visible in 20 frames and found in 4 of them: 4 / 20 = 20% < 25%, so it is deleted. If it is found in 12 (60%), it stays, provided at least three keyframes have observed it.
  3. Covisibility. Keyframes A and B share 40 points, A and C share 12 and B and C share 90. The covisibility graph has edges A–B (40) and B–C (90) but not A–C (12 < 15). The essential graph keeps B–C (90 is less than the 100 needed for a strong edge, so it is only kept if it is on the spanning tree). During loop correction only these sparse edges are optimised, so the cost scales with the number of keyframes and not the number of points.
  4. Keyframe culling. A keyframe sees 400 points. 370 of them are also seen by at least three other keyframes: 370 / 400 = 92.5% ≥ 90%, so the keyframe is redundant and is removed. The map shrinks but loses no coverage.
  5. Loop correction. After 300 m of monocular driving the scale has drifted by 5%: the trajectory returns 15 m short of where it started. The Sim(3) between the two keyframes has a scale of 1.05; spreading this correction over the essential graph pulls the whole loop closed, including the scale. A rigid SE(3) correction could not remove the scale drift.

Engineering implications and limitations

Engineering implications

  • Read the tracking state. OK, RECENTLY_LOST and LOST are surfaced by the system; log them with the number of tracked points. Most field failures start as a falling inlier count seconds before the state changes.
  • Tune the ORB parameters to the camera. The number of features (1000 to 2000), scale factor and levels, and the FAST thresholds all depend on the image resolution and texture.
  • Use a vocabulary suited to the environment. The default vocabulary is trained on generic images; in a very different setting (underwater, a greenhouse) place recognition can fail.
  • Expect loop closing to be slow and to move the map: downstream components (navigation, mapping) must tolerate a pose jump at a loop closure (see the TF discussion in M7-T2).
  • Save and reuse the map when working in a fixed environment (M7-T6); localisation-only mode reduces compute.
  • Prefer the visual-inertial variant for aggressive motion or low-texture scenes; the IMU bridges failures that a camera alone cannot.

Limits

Everything in the pipeline assumes distinctive static features. The architecture manages the consequences of tracking failures but does not remove them. Its sparse point map does not describe free space, so a robot that needs to plan paths must build an occupancy or mesh map from the same data (or from another sensor). And as a research system, it is not hardened for deployment: handling of camera restarts, time jumps and hardware faults is left to the integrator.

Common misconceptions

“ORB-SLAM tracks the previous frame.”

It tracks against a local map of points; the previous frame only provides the starting guess.

“More keyframes always improve accuracy.”

Redundant keyframes add compute without information; the system culls them.

“Loop closure fixes drift automatically in any case.”

It needs a recognised revisit and passes a geometric check. In a long corridor without revisits, drift remains.

“Relocalisation always recovers.”

It requires that the place is in the map and recognisable from the new view; otherwise the system stays lost or (ORB-SLAM3) starts a new map.

Engineering takeaways

  1. ORB-SLAM splits the work into tracking (per frame), local mapping (per keyframe) and loop closing (occasional), around a shared keyframe map.
  2. The covisibility graph defines local neighbourhoods; the essential graph makes loop correction cheap.
  3. Keyframes are inserted generously and culled strictly; new map points must prove themselves.
  4. Loop closure uses bag-of-words recognition, a Sim(3) estimate for monocular input, and essential-graph optimisation.
  5. ORB-SLAM2 adds stereo and RGB-D; ORB-SLAM3 adds an IMU, fisheye cameras and multi-map operation.

References and further reading

Primary papers and authoritative documentation.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

Why does tracking in ORB-SLAM match the current frame to the local map rather than only to the previous frame?

Explanation

Frame-to-frame tracking accumulates drift because each pose is relative to a noisy previous one. Matching to map points, which are refined by local bundle adjustment, anchors the pose to a consistent, optimised structure and keeps drift small. The previous frame only supplies the initial guess for the search.

Question 2

The reference keyframe has 200 map points and the current frame tracks 195 of them. Is a new keyframe inserted, and why?

Explanation

No: 195 / 200 = 97.5% is above the 90% threshold, so the view has hardly changed and a keyframe would add little. Insertion needs the frame to track fewer than 90% of the reference keyframe’s points (as well as the other conditions).

Question 3

Why does monocular ORB-SLAM compute a Sim(3) for a loop closure, but the stereo version can use SE(3)?

Explanation

A monocular system has an unobservable scale that drifts, so the two ends of a loop differ by a rotation, translation and a scale factor; the correction must include the scale. In stereo and RGB-D the scale is measured in each frame, so only a rigid transform is needed.

What’s next

Cameras give rich information but fail under fast motion and give no metric scale on their own. The next lesson adds an IMU and shows how the two sensors cover for each other.

Practice this lesson

Investigate ORB-SLAM Failure Modes

Given the state logs of a keyframe-based tracker, identify why tracking was lost and how it relocalised.

Next lesson

M6-T6 — Visual-Inertial Odometry

Explain IMU preintegration and why fusion fixes scale and fast motion.

Nothing below is required to finish this lesson.