M6-T1 — Camera Models and Image Features

Learning objectives

After this lesson, you should be able to:

• Write the pinhole projection with the intrinsic parameters f_x, f_y, c_x, c_y and compute the pixel of a 3D point.
• Apply a radial distortion model and say why SLAM needs calibrated, undistorted measurements.
• Explain what the Harris response measures and distinguish flat, edge and corner regions.
• Describe a binary descriptor (ORB) and say what makes a feature good for SLAM.

The problem and the intuition

Hold a finger in front of one eye. Its image sits at some position on your retina. From that position alone you know the direction of the finger but not its distance: the same image position is produced by a small finger close by and a large finger far away. A camera is the same. Each pixel is a ray, and depth along the ray is lost.

This is the opposite of a lidar, which measures the range and gives up very little else. For SLAM it has two consequences. First, the geometry is projective: lines stay lines but distances and angles do not, so the maths (epipolar geometry, M6-T3) differs from the rigid-body geometry of Modules 2 to 5. Second, we cannot match raw pixels on a large scale: a robot needs a few hundred distinctive, repeatable points that can be found again in another image. Those are image features.

A pixel is a ray. Depth is recovered only by seeing the same point from two places, so the first job of visual SLAM is to find the same point again.

The model: projection, distortion, corners and descriptors

Problem. Given the position of a 3D point in the camera frame, predict its pixel; given pixels in an image, choose points that can be found again in the next image.

Assumptions. The camera is a central projection (a single optical centre); the lens distortion is small and well modelled by a low-order radial polynomial; the scene is static and Lambertian enough that a point looks similar from nearby viewpoints; the intrinsics are calibrated and fixed.

The pinhole model

A point X_c = (X, Y, Z) in the camera frame (Z along the optical axis, pointing forward) first projects to normalised image coordinates x = X/Z, y = Y/Z, which carry the direction of the ray. The intrinsic matrix K converts them to pixels:

For a point in the world, first transform it with the camera pose: X_c = R^cw X_w + t^cw (M2-T1, M2-T2). Points with Z ≤ 0 are behind the camera and must be discarded.

Lens distortion

Real lenses bend the image. The common radial model scales the normalised coordinates before applying K:

A negative k₁ produces barrel distortion (straight lines bulge outwards, wide-angle lenses), a positive one pincushion. Tangential terms and, for fisheye lenses, different models (equidistant, Kannala–Brandt) exist. SLAM front ends either undistort the image once or undistort the feature locations and work in normalised coordinates.

Corners: the Harris response

A good place to track is one that looks different when shifted in any direction. Over a small window, build the structure tensor M = Σ [[I_x², I_xI_y], [I_xI_y, I_y²]] from the image gradients. Its two eigenvalues tell the story: both small, a flat region; one large, one small, an edge (it can slide along the edge); both large, a corner. Harris and Stephens (1988) score this with R = det M − κ (tr M)² with κ ≈ 0.04; Shi and Tomasi (1994) use the smaller eigenvalue min(λ₁, λ₂) directly.

Descriptors

A corner location is not enough to find it again; we need a signature of its surroundings. SIFT (Lowe, 2004) uses a 128-number histogram of gradients: robust but slow. ORB (Rublee et al., 2011) combines the FAST corner detector with a rotation-aware version of the BRIEF descriptor: 256 binary intensity comparisons between pairs of pixels in a patch, rotated by the patch’s dominant orientation. Two ORB descriptors are compared by their Hamming distance, the number of differing bits, which a processor computes in a few instructions. This is why ORB runs in real time on a CPU.

K

intrinsic matrix: focal lengths f_x, f_y in pixels and principal point (c_x, c_y).

(x, y)

normalised image coordinates (X/Z, Y/Z): the ray direction, free of the intrinsics.

k₁, k₂

radial distortion coefficients.

M, R

structure tensor of a window and the Harris corner response.

What makes a feature good for SLAM

A worked example: projecting a point, adding distortion, scoring corners

1. Projection. A camera has f_x = f_y = 600 px and principal point (320, 240) on a 640 × 480 image. A point is at (0.5, −0.2, 4) m in the camera frame. Normalised coordinates: x = 0.5 / 4 = 0.125, y = −0.2 / 4 = −0.05. Pixels: u = 600 × 0.125 + 320 = 395, v = 600 × (−0.05) + 240 = 210. The point is right of and above the image centre.
2. Field of view. The horizontal field of view is 2·atan(320 / 600) ≈ 56.1°. A point at 5 m laterally 3 m off the axis (angle 31°) is outside it.
3. With distortion. Take k₁ = −0.2 (mild barrel). r² = 0.125² + 0.05² = 0.018125, so the factor is 1 − 0.2 × 0.018125 = 0.996375. The distorted pixel is (320 + 600 × 0.125 × 0.996375, 240 − 600 × 0.05 × 0.996375) = (394.73, 210.11). Here the shift is only 0.3 px, but it grows with r²: at the image corner (x = 0.53, y = 0.4, r² = 0.44) the factor is 0.912 and the shift is about 35 px, so ignoring distortion would badly bias the geometry near the borders.
4. Harris. Three windows, with κ = 0.04. Flat: M = [[5, 0], [0, 4]], R = 20 − 0.04 × 81 = 16.8 (small). Edge: M = [[900, 0], [0, 10]], det = 9000, tr = 910, R = 9000 − 0.04 × 828 100 = −24 124 (large and negative). Corner: M = [[500, 100], [100, 400]], det = 190 000, tr = 900, R = 190 000 − 32 400 = 157 600 (large and positive). Only the last window is a good feature.
5. Hamming distance. Eight bits of two descriptors: 10110010 and 10010110. XOR gives 00100100, which has two set bits, so the distance is 2. For 256-bit ORB descriptors, same-point matches typically differ by a few tens of bits and unrelated points by about 128 (half of the bits).

Engineering implications and limitations

Engineering implications

• Calibrate once, carefully, and keep it fixed. Focal length, principal point and distortion enter every residual. A temperature-driven focus shift or a changed lens setting silently degrades accuracy (M6-T8).
• Work in normalised coordinates. Undistort feature positions and use the ray direction in the geometry; this makes the algorithms independent of the particular camera.
• Distribute features. Divide the image into a grid and keep the best few per cell. A cluster of features gives a poorly conditioned pose, in the same way that a corridor does for lidar.
• Match the detector to the compute budget. FAST/ORB for embedded, SuperPoint-style learned features on a GPU; learned features can survive illumination change that breaks ORB, at a cost.
• Respect the exposure. Auto-exposure changes that happen during motion break brightness constancy and can make features disappear (M6-T2, M6-T9).
• Know the global-shutter / rolling-shutter difference. A rolling shutter reads rows at different times, which acts like motion distortion in a lidar scan (M5-T2).

Limits

The pinhole model with low-order distortion fits most perspective lenses and breaks down for wide fisheye and catadioptric cameras, which need different models. Hand-designed features assume textured, well-lit, static scenes. White walls, glass, repeated patterns and night scenes defeat them, which is why the failure modes of M6-T9 are mostly failures of the feature front end.

Common misconceptions

Engineering takeaways

1. A pixel (u, v) is a ray; depth is lost and must be recovered from several views.
2. The projection is u = f_xX/Z + c_x, v = f_yY/Z + c_y, with radial distortion applied to the normalised coordinates first.
3. A good feature has two large structure-tensor eigenvalues: flat regions and edges cannot be localised.
4. ORB pairs the FAST corner with a binary descriptor compared by Hamming distance, which makes real-time matching cheap.
5. Features should be repeatable, distinctive, well localised, well distributed and static.

References and further reading

Primary papers and authoritative documentation.

• Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press.
• Harris, C. and Stephens, M. (1988). A combined corner and edge detector. Alvey Vision Conference.
• Shi, J. and Tomasi, C. (1994). Good features to track. IEEE CVPR.
• Rublee, E., Rabaud, V., Konolige, K. and Bradski, G. (2011). ORB: An efficient alternative to SIFT or SURF. IEEE ICCV.
• Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2).
• Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

With a camera model and a way to find distinctive points, the next lesson looks at following them from frame to frame: matching descriptors versus tracking with optical flow, and the checks that reject bad matches.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T10: Map Construction and Loop Closure

  Compare local map management, submaps and place recognition such as Scan Context for LiDAR loop closure.

• Lab
  M6-L1: Diagnose Feature-Tracking Failure

  Tracked features vanish or drift: diagnose blur, low texture and bad thresholds and repair the tracker.