M8-T7 — Neural and Implicit Mapping

Learning objectives

After this lesson, you should be able to:

• Write the volume-rendering equation and compute the weights of samples along a ray.
• Explain how a differentiable renderer lets a map and a camera pose be estimated from images.
• Compare implicit and Gaussian representations with TSDF volumes and point maps in memory, speed and quality.
• State what is established and what is still open in neural and implicit SLAM, with the primary references.

The problem and the intuition

Think of the map as a very good painter that can, on request, draw the room from any viewpoint. Given the camera’s pose, it draws the view; you compare the drawing with the photograph, and the difference tells you two things: how to adjust the painter’s knowledge of the room (mapping) and how to adjust the camera’s pose until the drawing matches (tracking). Both adjustments are gradient descent on the same image-difference loss, which requires that the drawing process be differentiable.

The painter can be a small neural network queried at 3D points (a radiance field), a network plus a grid of learned features (faster), or a cloud of coloured, semi-transparent 3D Gaussians that are projected to the image and blended (very fast, explicit). All three need the same ingredient: a smooth rendering equation along each camera ray. What they buy is dense, photorealistic geometry and appearance with a compact description; what they cost is training time, memory on a GPU, and the loss of the guarantees and the interpretability of classical maps.

Neural and Gaussian maps turn mapping and tracking into fitting a renderer to images. The renderer is differentiable, so gradients tell both the map and the camera pose how to improve.

The mechanism, the systems, the status

Problem. From a stream of images (with or without depth), estimate the camera trajectory and a representation from which novel views and depth can be rendered, online.

Assumptions. A mostly static scene with photometric consistency; a GPU; for monocular RGB, enough view overlap and parallax; for RGB-D, depth that is reasonably accurate; known camera intrinsics.

Volume rendering (the shared ingredient)

Along a ray r(s) = o + s d, sample N points with spacing δ_i. At each, the representation gives a density σ_i and a colour c_i. The opacity of the segment is α_i = 1 − exp(−σ_i δ_i) and the transmittance (probability that the ray has not been stopped before sample i) is T_i = Π_j<i (1 − α_j). The pixel colour and the expected depth are:

The weights w_i = T_iα_i sum to at most 1. The same formula is used for NeRF (Mildenhall et al., 2020) and, with the Gaussians projected and sorted by depth and blended front to back, for 3D Gaussian Splatting (Kerbl et al., 2023).

Representations

Tracking and mapping with a renderer

1. Tracking. Fix the map. Render the current view from the predicted pose; compare with the image; take gradient steps on the pose (6 parameters on SE(3)) to reduce the photometric and depth loss.
2. Mapping. Fix the poses (of a window of keyframes). Add new representation elements where the scene is new (new Gaussians, new grid cells); take gradient steps on the map parameters.
3. Keyframes and scheduling. As in classical systems, only keyframes drive mapping, with a sliding window and a replay of old keyframes to avoid forgetting.

σ, α, T

density, opacity per segment, and transmittance along a ray.

NeRF

neural radiance field: an MLP mapping a 3D point and direction to density and colour.

3DGS

3D Gaussian Splatting: a scene as a set of anisotropic 3D Gaussians with opacity and colour.

TSDF

truncated signed distance function stored on a voxel grid.

Established versus open

A worked example: rendering weights and memory

1. Weights along a ray. Three samples with spacing δ = 0.1 m and densities σ = 0.5, 2.0 and 5.0. Opacities: α₁ = 1 − e^−0.05 = 0.0488, α₂ = 1 − e^−0.2 = 0.1813, α₃ = 1 − e^−0.5 = 0.3935. Transmittances: T₁ = 1, T₂ = 1 − 0.0488 = 0.9512, T₃ = 0.9512 × (1 − 0.1813) = 0.7788. Weights w = T α: 0.0488, 0.1724, 0.3064. They sum to 0.528: the ray has been 52.8% absorbed within these samples, and the remaining 47.2% would go to samples further along (or the background).
2. The pixel. If the three colours are (0.2, 0.2, 0.2), (0.8, 0.1, 0.1) and (0.1, 0.8, 0.1), the red channel contribution is 0.0488 × 0.2 + 0.1724 × 0.8 + 0.3064 × 0.1 = 0.0098 + 0.1379 + 0.0306 = 0.178. The rendered colour depends smoothly on every σ and c, so a gradient of the loss with respect to them exists: that is the whole trick.
3. Memory: dense TSDF. A 5 m × 5 m × 5 m room at 1 cm voxels has 500³ = 1.25 × 10⁸ voxels. At 8 bytes per voxel (TSDF, weight, colour) that is 1 GB. Voxel hashing (only allocating near surfaces) reduces it by about 10 to 20 times in real scenes. A Gaussian-splatting model of a room typically has 10⁵ to 10⁶ Gaussians at about 59 floating-point numbers each (position 3, scale 3, rotation 4, opacity 1, spherical-harmonic colour 48) ≈ 236 bytes: 24 MB to 240 MB. These are order-of-magnitude figures, to be measured in each system.
4. What the loss does not see. A mirror or a glass wall: the renderer fits the reflected scene as geometry behind the mirror, and the photometric loss is minimal. Rendering quality is high, and the map is wrong for a robot that must avoid the wall. This is why photometric agreement is not geometric accuracy.
5. Tracking by rendering. A pose error of 5 cm shifts a surface 2 m away by about 2.5% of its distance in the image: the rendered image is shifted by a few pixels, and the L1 loss rises. The gradient points to the correction. If the motion is large or blurred, the loss surface has local minima (as in M6-L1, for the same reason), and the tracker needs a good initial guess from a motion model or an IMU.

Engineering implications and limitations

Engineering implications

• Treat it as a mapping layer on top of tracking you trust. Using classical SLAM or visual-inertial odometry for the poses and a neural or Gaussian map for appearance is the pragmatic combination today.
• Check geometry independently. Evaluate depth and surface accuracy against ground truth, not only image quality (PSNR, SSIM).
• Budget GPU memory and time, and confirm real-time claims on your hardware: many reports use high-end desktop GPUs.
• Expect fragility in tracking: motion blur, exposure changes, dynamic objects and textureless areas all break photometric assumptions (M6-T9).
• Plan for forgetting and scale: online training can forget old regions; large scenes need submaps and loop closure that most of these systems lack.
• Do not feed it to a planner without a geometric check. Rendering can be photoreal and metrically wrong.

Limits

Most results come from small indoor scenes with RGB-D cameras and short sequences. Loop closure, global consistency, large scale, dynamic scenes and uncertainty are open, and the field changes fast: specific accuracy and speed numbers in papers go out of date within months. There is no established theory that explains when the photometric loss leads to a correct map. Read each new system for its data, depth input, scene size, hardware and evaluation protocol before relying on a headline number (M8-T8).

Common misconceptions

Engineering takeaways

1. Volume rendering C = Σ T α c is the shared mechanism; it is differentiable, so both the map and the camera pose can be fitted to images.
2. NeRF-style, feature-grid and Gaussian representations trade training speed, memory, editability and geometric accuracy.
3. Established: offline fitting of a static scene to posed images. Demonstrated: online RGB-D systems on small indoor scenes. Open: monocular, large-scale, dynamic, uncertainty, loop closure.
4. Rendering quality is not geometric accuracy: verify depth and surfaces separately.
5. Use neural maps as an appearance layer on top of trusted tracking until the open problems are solved.

References and further reading

Primary papers. The status table in the text separates what is established from what is demonstrated only in research systems and what is open.

• Mildenhall, B. et al. (2020). NeRF: Representing scenes as neural radiance fields for view synthesis. European Conference on Computer Vision (ECCV).
• Sucar, E., Liu, S., Ortiz, J. and Davison, A. J. (2021). iMAP: Implicit mapping and positioning in real-time. IEEE ICCV.
• Zhu, Z. et al. (2022). NICE-SLAM: Neural implicit scalable encoding for SLAM. IEEE CVPR.
• Wang, H., Wang, J. and Agapito, L. (2023). Co-SLAM: Joint coordinate and sparse parametric encodings for neural real-time SLAM. IEEE CVPR.
• Müller, T., Evans, A., Schied, C. and Keller, A. (2022). Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (SIGGRAPH) 41(4).
• Kerbl, B., Kopanas, G., Leimkühler, T. and Drettakis, G. (2023). 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4).
• Keetha, N. et al. (2024). SplaTAM: Splat, track and map 3D Gaussians for dense RGB-D SLAM. IEEE CVPR.
• Matsuki, H., Murai, R., Kelly, P. H. J. and Davison, A. J. (2024). Gaussian splatting SLAM. IEEE CVPR.
• Newcombe, R. A. et al. (2011). KinectFusion: Real-time dense surface mapping and tracking. IEEE ISMAR.
• Curless, B. and Levoy, M. (1996). A volumetric method for building complex models from range images. SIGGRAPH.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

With so many systems and so much recent progress, claims need careful measurement. The next lesson is about evaluating SLAM correctly and reading benchmark protocols.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T6: Learning-Based SLAM

  Survey learned components: features, depth, place recognition, differentiable SLAM and end-to-end odometry.

• Lesson
  M5-T11: Dense Map Representations

  Compare occupancy octrees, signed distance fields and surfels, fuse range measurements into a TSDF, and say what a loop closure does to a dense map.

• Lab
  M8-L4: Evaluate a SLAM System Under Controlled Failures

  Inject controlled failures (noise, outliers, dropouts) into a SLAM run and report degradation honestly.