P8-T1 — Why Robots Combine Multiple Sensors

Learning objectives

After this lesson, you should be able to:

• Distinguish redundancy, complementarity and coverage as reasons to combine sensors.
• Compute the variance gain from combining independent estimates and the effect of correlated errors.
• Compare AND and OR combination of detections in terms of detection and false alarm probabilities.
• State the conditions - calibration, synchronisation, common frame, known uncertainty - on which fusion is valid.

The idea: sensors with different weaknesses

Every sensor has conditions in which it fails: a camera at night or in glare, a lidar in heavy rain or on a black mirror-like surface, radar on small plastic objects. If two sensors fail in different conditions, then in most situations at least one of them is working, and where both work they check each other. That is the core argument for combining them.

Redundancy

Two sensors measuring the same thing allow detection of a fault: if they disagree by more than their noise allows, something is wrong and the system can react, by slowing down or by handing over to a backup. Redundancy also allows the robot to continue when one sensor is lost. It is the basis of safety arguments, which is why it is useful even when it does not improve accuracy.

Complementarity

Sensors that are good at different things combine into something better than each. A camera gives the direction of an object precisely and its distance poorly; a lidar gives the distance precisely and the direction less well. Their combination knows both. The next lessons take this up in detail, and Lab P8-L4 computes it.

Coverage

Sensors differ in field of view, range, resolution and operating conditions. A wide-angle camera sees the whole front but few details at a distance; a narrow telephoto camera or a long-range lidar sees far ahead. Thermal cameras work in darkness and radar in fog. The system hands over between them or combines them where they overlap.

What fusion needs

The benefits are only available under four conditions, and the lessons of this module are about them. The sensors must be calibrated against each other, so that a point in one is expressed correctly in the other (Lesson P8-T3). They must be synchronised, so that the data being combined describe the same moment. They must report in a common frame, with known units and conventions. And their uncertainties must be known, so that each is trusted according to its accuracy (Lesson P8-T5). If any of these fails, the combined result is not a better version of the inputs, it is an average of inconsistent facts that looks confident.

The mathematics: what combining can and cannot gain

Independent estimates. Two independent unbiased estimates of the same quantity with variances σ₁² and σ₂² combine, with the best weights, into one whose variance is

σ₁², σ₂²

the variances of the two estimates, in the squared units of the quantity.

σ²

the variance of the combined estimate. It is below both inputs: with σ₁ = σ₂ = σ it is σ²/2, a standard deviation of 0.71σ.

The weights are proportional to the inverse variances, so the more accurate sensor dominates. If the accuracies are very different the gain is small: a sensor with ten times the standard deviation improves the better one by under 1% in standard deviation. Combining is valuable when the sensors are comparably accurate in the same direction, or accurate in different directions.

Correlated errors. If the errors have correlation coefficient ρ (they share a cause: the same vibration, the same clock, the same light), the plain average of two equal-variance sensors has variance

ρ

the correlation of the two sensors’ errors, from 0 (independent) to 1 (identical).

σ²

the variance of each sensor.

At ρ = 0 the variance is σ²/2; at ρ = 0.5 it is 0.75σ²; at ρ = 1 it is σ², no gain at all. A second sensor of the same type, mounted next to the first, has a large ρ for anything that depends on the environment.

Combining decisions. Suppose two independent detectors each find an object with probability p_d and raise a false alarm with probability p_fa per cell. The OR rule reports an object if either detects it; the AND rule requires both:

OR raises detection and also false alarms; AND lowers false alarms and also detection. Which is right depends on which error costs more (Lesson P6-T6).

Worked examples: gains and their limits

Same accuracy. Two sensors each with σ = 0.5 m, independent. 1/σ² = 4 + 4 = 8, so σ = 0.354 m, a 29% reduction.

Complementary accuracy. At 10 m a size-based camera estimate of range has σ = 0.9 m and a lidar has σ = 0.05 m (Lab P8-L2). Fused along the line of sight: 1/σ² = 1/0.81 + 1/0.0025 = 1.23 + 400 = 401, so σ = 0.0499 m. The camera adds almost nothing along the line of sight, as expected, but across it (camera σ = 0.1 m from the bearing, lidar σ = 0.2 m) the fusion gives σ = 0.089 m, better than both. The gain is in a direction, not in general.

Correlation. Two cameras next to each other see the same glare. If ρ = 0.5 the average of their errors has variance 0.75σ², a standard deviation of 0.87σ: 13% better, against 29% for independent errors.

AND and OR. Each of two sensors detects with p_d = 0.9 and false-alarms with p_fa = 0.05 per cell.

OR lifts detection from 0.90 to 0.99 and nearly doubles the false alarms (0.0975); AND cuts false alarms twentyfold and loses nine points of detection. Neither is free, and the choice of rule is the choice of what to protect.

A failure of the prerequisites. Suppose the camera is rotated by 3° relative to the stored calibration. At 10 m the lateral error of a camera point is 10 × tan 3° = 0.52 m. If the lidar says 0.2 m accuracy and the camera 0.1 m, an inverse-variance fusion weights the camera 80% across the line of sight, so the fused position inherits most of the 0.52 m error, with a small stated uncertainty. The fusion has made the result worse than the lidar alone, while reporting a better covariance.

Engineering implications and limitations

Name the reason for each sensor. For every sensor in the design write whether it is there for redundancy, complementarity or coverage, and what condition it is meant to cover. A sensor with no reason is cost and a source of faults.

Choose sensors with different failure modes. Two cameras share the failure of darkness; a camera and a radar do not. Diversity matters more than the number.

Check independence. Look for shared causes: common power, a common clock, a common mounting (vibration), the same weather. Treat correlated sensors as one sensor for the safety argument.

Do the prerequisites before the fusion. Calibrate, synchronise, fix the frame conventions, and measure each sensor’s noise before writing any fusion code. A fusion built on guesses cannot be debugged, because every error could come from any of them.

Always keep a single-sensor fallback. Compare the fused output with each sensor alone on logged data. If the fusion is worse than the best single sensor in some condition, you have a bug or a broken assumption, and you must know about it.

Monitor disagreement. The rate at which sensors disagree beyond their uncertainty is a cheap and powerful health indicator. A sudden rise means a loosened mount, a clock fault or an environment change (Lesson P8-T5).

Stay honest about the labs. In this module all sensor data are simulated with stated error models. They show how fusion behaves under those models; they do not describe any real sensor.

Common misconceptions

Engineering takeaways

1. Sensors are combined for redundancy, complementarity or coverage; each has a different benefit and a different design implication.
2. Independent estimates combine as 1/σ² = 1/σ₁² + 1/σ₂²: equal sensors give 0.71σ, very unequal ones almost nothing, and correlated errors reduce the gain to σ²(1 + ρ)/2.
3. OR combination raises detection and false alarms (0.90 → 0.99, 0.05 → 0.0975); AND lowers both (0.81, 0.0025).
4. Fusion is valid only if the sensors are calibrated, synchronised, in a common frame and have known uncertainties; otherwise it makes results worse while looking better.
5. Compare the fused result with each sensor alone, and monitor disagreement in operation.

References and further reading

Textbooks and a handbook chapter. The sensor accuracies, probabilities and the 3° miscalibration in the examples are illustrative.

• Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: combining measurements with their uncertainties.
• Bar-Shalom, Y., Li, X.-R. and Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation. Wiley: fusion of estimates and the effect of correlated errors.
• Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D. (2011). Introduction to Autonomous Mobile Robots, 2nd ed. MIT Press: sensors for mobile robots and their error characteristics.
• Durrant-Whyte, H. and Henderson, T. C. (2016). Multisensor Data Fusion. In Springer Handbook of Robotics, 2nd ed.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the two sensors used throughout this module, the camera and the lidar, and where each is strong and weak.

Related content

Nothing below is required to finish this lesson.

• Before
  P7-T6: Occlusion, Identity Switches and Missed Detections

  Diagnose and reduce the main tracking failures.

• Lab
  P8-L1: Combine Camera Detections with Depth Information

  Lift 2D detections to 3D positions using depth and a camera model.