FixTheRobot Explore Robotics / Systems Engineering / Engineering Guides / Module 02

Engineering Guides · Module 02

Sensor combination: camera + LiDAR

The camera says what something is; the LiDAR says where it is. Fusing them gives labelled 3D objects.

  • Advanced
  • Module 02: Sensor Selection, Trade-offs & Integration
  • Written for ROS 2 Jazzy where ROS 2 is used
  • Sources checked 2026-11-05

Assumptions: General fusion guidance. Named algorithms appear only where their READMEs were checked (see the SLAM Selection Guide). No performance figures are given.

1. Engineering problem

A camera detects and classifies but cannot measure distance directly; a LiDAR measures distance but has little appearance information. Perception needs both, aligned in space and time.

2. When is this needed?

  • Obstacle detection and classification.
  • Colouring maps.
  • Object-level perception for planning.

3. Selection and design decisions

What each sensor contributes

SensorContributionWeakness the other covers
CameraClassification, texture, colourNo direct range; light needed
LiDARAccurate 3D position of the surfaceSparse at range, no appearance

Fusion approaches

  • Project LiDAR points into the image and attach detections to 3D clusters (early or object-level fusion).
  • Detect in 3D and verify with the image (late fusion).
  • Depth completion or learned fusion (needs training data).

4. Advantages and disadvantages

  • Early fusion uses more information and is sensitive to calibration; object-level fusion is robust and loses detail.
  • Needs tight camera–LiDAR extrinsics and time alignment.

5. Requirements and constraints

  • Time synchronisation to within what the estimator tolerates (use the Sensor Time Synchronization Simulator).
  • Calibrated extrinsics between every fused pair, with the transform direction documented (see the frame-conventions guide).
  • Extrinsics validated with a reprojection check.
  • Time offset small relative to platform motion (use the sync simulator).
  • Intrinsics and distortion calibrated.

6. Architecture

camera ─▶ 2D detections ─┐
                         ├─▶ association (project / cluster) ─▶ labelled 3D objects
LiDAR ─▶ clusters ───────┘

7. Implementation

  1. Calibrate intrinsics, then camera–LiDAR extrinsics; document the direction (see the frame guide).
  2. Validate by projecting LiDAR points onto a target.
  3. Synchronise or measure the time offset.
  4. Associate detections with clusters and measure association accuracy.

Compare structures in the Sensor Fusion Architecture Builder.

8. Real examples

  • An autonomous shuttle that labels pedestrians with distances.

9. Failure scenarios

  • Extrinsic error: points miss the object (see the misalignment diagnostic).
  • Time offset: alignment only fails when moving.
  • Occlusion: the camera sees what the LiDAR cannot, and vice versa.
  • Glass and reflective surfaces appear in one sensor only.

10. Troubleshooting

  1. Use the Camera–LiDAR Projection Visualizer to identify the error pattern.
  2. Turn the robot in place to test time offset.
  3. Check intrinsics at the image edges.

11. Validation

  • Projected points land on a target within your pixel tolerance at three distances.
  • Alignment holds while turning at the maximum rate.
  • Association accuracy measured on labelled data.

12. Engineering checklist

13. Further references

  • Class-level guidance written from general engineering knowledge; it contains no product specifications. Product-specific numbers must come from the manufacturer’s datasheet for the exact model and operating mode.
  • ROS 2 message types: sensor_msgs (jazzy branch).

Related on FixTheRobot