Skip to the lesson
FixTheRobot Explore Robotics / Perception / P8 Sensor Fusion and Scene Understanding / P8-T2

Perception · Theory · P8

P8-T2 — Complementary Strengths and Weaknesses of Cameras and LiDAR

A camera knows what and which way. A lidar knows how far. Neither knows everything.

  • Lesson 2 of 6
  • Module 8: Sensor Fusion and Scene Understanding
  • Theory
  • About 9 minutes to read

Cameras and lidars are the two sensors this module combines, and they are close to opposites. A camera records a dense, coloured picture of the scene and says nothing about distance; a lidar measures distance directly and returns a sparse set of points without colour. Where each is accurate, where it degrades, and why, are the facts every fusion decision rests on. This lesson puts numbers on them: how far apart lidar points are at a distance, how many pixels an object has, how fast depth error grows for a stereo pair, and what weather and surfaces do to each. Labs P8-L1 and P8-L2 use these models.

Learning objectives

After this lesson, you should be able to:

  • Compare what a camera and a lidar measure, and the failure conditions of each.
  • Compute lidar point spacing and camera pixel footprint of an object at a given distance.
  • Use the stereo depth error model and compare it with lidar accuracy over range.
  • Explain how lighting, weather and surface reflectance affect each sensor and what this implies for fusion.

The idea: appearance against geometry

A camera measures light: how much of it comes from each direction. From the pattern of light you can recognise things, read signs and separate objects by colour and texture. Distance has to be inferred, from size, from perspective, from two views or from learned cues. A lidar measures time of flight: it sends a pulse in a known direction and times the return, so each point comes with a direct, accurate distance. What it lacks is appearance: it sees the shape of a person but not the colour of their jacket, and its points are far apart.

PropertyCameraLidar
Measureslight intensity and colour per directionrange per direction by time of flight
Densitymillions of pixels, a regular gridtens to hundreds of thousands of points, sparse at range
Distanceinferred; error grows with rangedirect; error roughly constant (centimetres)
Semanticsrich: class, colour, text, lightsshape and sometimes reflectance only
Lightingdepends on it: dark, glare, shadowsindependent of ambient light (it carries its own)
Weatherdegrades in rain on the lens, fog; works in light raindegrades in heavy rain, fog, snow, dust (returns from droplets)
Surfaceshard cases: mirrors, glass, texturelesshard cases: black, wet, glass, mirrors (weak or no return)
Cost and sizecheap and smallmore expensive, larger, with moving parts or solid state

Complementary, not interchangeable

Almost every row of the table has a cell where one sensor is strong and the other weak. A camera can tell a pedestrian from a post; a lidar can tell how far each is. At night the lidar still works and the camera does not; in thick fog the reverse can hold for a thermal camera. This is the reason for combining them, and also the reason for care: the cases where one fails are the cases where you most need the other, and the fusion has to know which one to believe.

Why resolution depends on distance

Both sensors sample the world at fixed angles, so the distance between samples on an object grows with range. A lidar returns a point every fraction of a degree; a camera pixel covers a fraction of a degree too. Far objects are covered by few points and few pixels, which limits recognition for both. A camera has far more samples per object than a lidar at any range, and so wins on small and distant object recognition, but not on distance.

Depth error against distance for stereo and lidarA plot of depth error in metres against distance. A stereo camera pair with a 0.2 metre baseline and a focal length of 1000 pixels has an error that grows with the square of the distance, reaching 0.5 metres at 20 metres and over 3 metres at 50 metres. A lidar has a nearly constant error of 0.03 metres, shown as a flat line.010203040506000.71.42.12.83.5distance (m)depth error (m) orange: stereo, σ_z = z² σ_d / (f B)f = 1000 px, B = 0.2 m, σ_d = 0.25 px blue: lidar, about 0.03 m at any range At 20 m the stereo error is 0.5 m,at 50 m it is 3.1 m.
Depth error of a stereo camera grows with the square of distance; a lidar’s stays near a few centimetres. The curves cross at a few metres; beyond that, lidar is the better source of distance.

The mathematics: spacing, footprint and depth error

Lidar point spacing. A lidar with angular step Δθ between neighbouring beams (horizontally or vertically) places points a distance apart of about

spacing between lidar points at range r
s=r⋅Δθs=r\cdot \Delta \theta
r
the range to the surface in metres.
Δθ
the angular resolution in radians (0.2° = 0.00349 rad).
s
the distance between neighbouring points on a surface facing the sensor. On an oblique surface it is larger by 1/cos of the incidence angle.

The number of points on an object of width w is about w/s = w/(r Δθ) horizontally, and falls as 1/r; the count over the face of an object falls as 1/r².

Camera footprint. A pinhole camera of focal length f (in pixels) images an object of width w at distance r onto

pixels across an object
npixels=f⋅wrn_{\text{pixels}}=\frac{f\cdot w}{r}

This is the relation of Lesson P3-T1, and it is the same 1/r dependence as the lidar’s count, but with f of order 1000 px against a lidar’s 1/Δθ of order 300 points per radian.

Stereo depth error. A stereo pair with baseline B and focal length f measures depth z = f B / d from the disparity d in pixels. An error σd in disparity gives

depth error of a stereo pair
σz=z2 σdf B\sigma _{z}=\frac{z^{2}\,\sigma _{d}}{f\,B}
σd
the standard deviation of the disparity measurement in pixels (a quarter of a pixel is typical of good matching).
f, B
the focal length in pixels and the baseline in metres: a longer baseline and focal length lower the error.
z²
the reason depth error grows quadratically with range.

A lidar’s range error σL depends on the pulse timing and is nearly independent of range, a few centimetres.

Reflectance and range. For a flat surface of reflectivity ρ, the returned power falls as ρ/r². A lidar that needs a fixed minimum power therefore reaches a maximum range proportional to √ρ (a simplified model; real devices also saturate and have a minimum range).

Worked examples: points, pixels and error

Lidar points on a pedestrian. A lidar has Δθ = 0.2°, so s = r × 0.00349. At 20 m the spacing is 0.070 m; a 0.5 m wide person is covered by 0.5 / 0.070 ≈ 7 points across. At 50 m the spacing is 0.175 m, and the person gets about 3 points across.

Camera pixels on the same person. With f = 1000 px at 50 m the person is 1000 × 0.5 / 50 = 10 pixels wide. That is more samples than the lidar gives (3), but the camera cannot say how far the person is except by guessing.

Stereo depth error. With f = 1000 px, B = 0.2 m, σd = 0.25 px, f B = 200:

Distance zσz = z² × 0.25 / 200Lidar σ
5 m0.031 m0.03 m
20 m0.50 m0.03 m
50 m3.1 m0.03 m

The two have the same error at 5 m, a factor of 17 apart at 20 m and over 100 apart at 50 m. Beyond a few metres, distance should come from the lidar, or from the camera’s size cues (Lab P8-L2), whose error grows only in proportion to range.

Dark objects. A black surface reflects perhaps 10% of the light, against 80% for a white one. The maximum detection range scales with √ρ: √(0.1/0.8) = 0.35. A lidar that sees white targets at 100 m sees black ones at about 35 m. A dark-clothed pedestrian by night is a worse target for a lidar than a car with a bright number plate, and the camera cannot make up for it in the dark.

What fusion gains here. The camera finds and classifies the pedestrian at 50 m; the lidar, with 3 points, confirms an object and gives its distance to centimetres once it is near enough to hit it. At night the camera does not find it at all, and the lidar sees only a shape: a fusion has to rely on the lidar alone and say so.

Engineering implications and limitations

Know the error laws, not only the specification sheet. Specifications give the best case. Build the table of error against range, lighting and reflectance for your own sensors, and use it in the fusion (Lesson P8-T5).

Match the sensor to the question. Distance and shape: lidar. Class and appearance: camera. Speed directly: radar. Darkness: lidar, radar or thermal. Do not pick a sensor for the question it answers badly.

Plan for the worst surfaces. Test with black, wet, glass and metallic objects, and with strong sunlight on the lens. The robot will meet them, and the sensor’s failure on them is silent: no return looks like free space.

Treat “no return” carefully. A lidar that returns nothing in a direction has either seen empty space or failed to see a dark or specular object. A map that marks it as free is making a claim the sensor cannot support; mark such cells unknown unless there is positive evidence.

Remember weather. Rain, fog, snow and dust create false returns for lidar and wash out cameras. Log the weather and evaluate by condition (Lesson P6-T6).

Mind mounting. The two sensors see different parts of the scene from different positions. Parallax makes an object near the sensors appear at different directions in each; fusion at short range needs the extrinsic calibration to be good (Lesson P8-T3).

Keep the labels honest. The sensor models in the labs (a depth error of 0.012 z², a lidar of 0.05 m) are stated simulations chosen to show the effects, not the specification of any device.

Common misconceptions

“Lidar is just a better depth camera.”

It measures a different quantity, by time of flight, at a different density and without appearance. It fails on dark and specular surfaces and in heavy precipitation.

“Cameras cannot measure distance.”

Stereo, motion, size cues and learned depth all give distance. The point is that the error grows quickly with range for most of them.

“A lidar sees everything in its field of view.”

It samples at fixed angles, so small objects at range may be missed between beams, and absorbing or specular surfaces may return nothing.

“More pixels or more beams solves it.”

Higher resolution helps recognition and spacing but does not change the physics of light, reflectance and weather.

Engineering takeaways

  1. A camera measures appearance per direction; a lidar measures range per direction. Their strengths and weaknesses are largely complementary.
  2. Both sample by angle: lidar spacing is r·Δθ (0.07 m at 20 m for 0.2°), a camera gives f·w/r pixels across an object.
  3. Stereo depth error grows as z²σ_d/(fB): 0.03 m at 5 m, 0.5 m at 20 m, 3.1 m at 50 m for the example; lidar stays near 0.03 m.
  4. Lidar range falls with dark surfaces (about √ρ), weather and specular materials; camera performance falls with low light and glare.
  5. No return is not free space: treat missing lidar returns as unknown unless evidence says otherwise.

References and further reading

Textbooks and datasets for sensor behaviour. The sensor parameters (0.2° resolution, f = 1000 px, 0.2 m baseline, quarter-pixel disparity error) are illustrative round numbers, not those of a particular device.

  • Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: stereo geometry and depth error.
  • Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: triangulation and its error.
  • Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D. (2011). Introduction to Autonomous Mobile Robots, 2nd ed. MIT Press: ranging sensors and their characteristics.
  • Geiger, A., Lenz, P. and Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. CVPR: a camera plus lidar dataset.
  • Caesar, H. et al. (2020). nuScenes: A multimodal dataset for autonomous driving. CVPR: camera, lidar and radar data under varied conditions.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A lidar has 0.1° angular resolution. How far apart are neighbouring points at 30 m, and how many fall across a 0.4 m wide object?

Explanation

s = 30 × 0.1 × π/180 = 0.0524 m; 0.4 / 0.0524 ≈ 7.6, about 7 or 8 points across.

Question 2

A stereo pair has f = 800 px, B = 0.3 m and σd = 0.3 px. What is the depth error at 15 m, and at what distance does it exceed 1 m?

Explanation

σz = 15² × 0.3 / (800 × 0.3) = 225 × 0.3 / 240 = 0.28 m. It reaches 1 m when z² = 1 × 240 / 0.3 = 800, z = 28 m.

Question 3

A lidar returns nothing in a patch of the floor in front of a robot. Give two possible causes and say how the map should treat the patch.

Explanation

Either the floor is truly empty out to the maximum range (free space), or the surface is black, wet, glass-like or otherwise specular and returned too little light. The map should mark it unknown unless another sensor or an earlier view confirms free space, and the planner should treat unknown cells cautiously.

What’s next

Next: the prerequisite of every fusion, alignment: calibration, coordinate frames and timestamps.

Practice this lesson

Combine Camera Detections with Depth Information

Lift 2D detections to 3D positions using depth and a camera model.

Next lesson

P8-T3 — Coordinate Alignment and Timestamps

Align measurements with extrinsics and handle time offsets.

Nothing below is required to finish this lesson.