FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 02

Engineering Guides · Module 02

Sensor combination: camera + LiDAR + radar

Three modalities with different weaknesses: appearance, geometry and weather-proof velocity. Redundant perception.

  • Advanced
  • Module 02: Sensor Selection, Trade-offs & Integration
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-05

Assumptions: General fusion guidance. Named algorithms appear only where their READMEs were checked (see the SLAM Selection Guide). No performance figures are given.

1. Engineering problem

Cameras fail in the dark and glare, LiDAR in heavy rain, fog and dust, radar lacks detail. Combining them keeps perception working across conditions and gives a cross-check.

2. When is this needed?

  • Outdoor autonomy across weather and lighting.
  • Applications that need redundant perception.

3. Selection and design decisions

What each sensor contributes

SensorContributionWeakness the other covers
CameraClassification and appearanceLight, glare, no range
LiDARPrecise 3D shapeWeather, glass
RadarDetection and radial velocity in bad weatherCoarse, ghost targets

Fusion approaches

  • Object-level (late) fusion with a tracker that weights each sensor by condition.
  • Early fusion of radar points with camera and LiDAR features (more calibration-sensitive).
  • Cross-checking: an object seen by only one modality is flagged.

4. Advantages and disadvantages

  • Three calibrations, three clocks and three failure modes to manage.
  • Object-level fusion is robust; early fusion extracts more but needs accurate extrinsics.
  • Cost and compute are the highest of the combinations.

5. Requirements and constraints

  • Time synchronisation to within what the estimator tolerates (use the Sensor Time Synchronization Simulator).
  • Calibrated extrinsics between every fused pair, with the transform direction documented (see the frame-conventions guide).
  • Time synchronisation across three streams.
  • Extrinsics for all pairs, including radar boresight.
  • Regulatory approval for the radar band; radome material.

6. Architecture

camera ─▶ detections ─┐
LiDAR  ─▶ clusters ───┼─▶ association + tracker (condition-aware weights) ─▶ tracked objects
radar  ─▶ targets ────┘

7. Implementation

  1. Calibrate each pair and document directions.
  2. Synchronise clocks and measure remaining offsets.
  3. Build the association and tracking; define which modality dominates in each condition.
  4. Test in rain, fog, dust, low light and glare.

Compare structures in the Sensor Fusion Architecture Builder.

8. Real examples

  • An all-weather outdoor shuttle.

9. Failure scenarios

  • Radar ghost targets from metal surfaces.
  • Camera glare and LiDAR rain returns at the same moment.
  • Association errors from timing offsets.
  • Common-mode failures (power, shared clock).

10. Troubleshooting

  1. Compare detections per modality on the same recording.
  2. Check association distances against calibration error.
  3. Log which modality was trusted at each time.

11. Validation

  • Detection of people and static objects verified in each condition.
  • Deliberate loss of each modality leaves the others functional.
  • False alarm rate below your limit.

12. Engineering checklist

13. Further references

  • Class-level guidance written from general engineering knowledge; it contains no product specifications. Product-specific numbers must come from the manufacturer’s datasheet for the exact model and operating mode.
  • ROS 2 message types: sensor_msgs (jazzy branch).

Related on FixTheRobot