FixTheRobot Explore Robotics / Systems Engineering / Troubleshooting

Troubleshooting · Sensors

Camera and LiDAR detections do not align

Projected LiDAR points land beside the object in the image. Find out whether the error is constant, depends on distance, or depends on motion; each points to a different cause.

  • Related modules: 06, 08
  • ROS 2 Jazzy

1. Symptoms

  • LiDAR points projected into the image sit consistently left, right, above or below the objects they hit.
  • The offset grows with distance, or is the same at every distance.
  • Alignment is fine when the robot is still and bad when it moves or turns.
  • Alignment was good yesterday and is wrong today.

2. Possible causes

  • Transform direction or axis mix-up (TCL vs TLC, optical vs body frame): large, obvious error from the start.
  • Rotation error in the extrinsics: a shift that is about the same in pixels at every distance.
  • Translation error in the extrinsics: an offset that shrinks with distance (large for near objects).
  • Time offset or latency between the two sensors: error that appears only with motion and grows with turn rate or speed.
  • Wrong intrinsics or distortion: error that varies across the image, worst near the edges.
  • Mechanical change: a loosened mount or a knocked sensor after a collision.

3. Measurements to collect

  • The pixel offset of projected points on a few clear objects at several distances (near, mid, far), with the robot still.
  • The same offset with the robot turning in place at a known rate.
  • The measured mounting translation (tape measure or CAD) versus the value in the transform.
  • Message stamps and receive times for both sensors (see the bag-latency guide).
  • The determinant and orthonormality of the rotation in use.

4. Diagnostic commands

Commands are not executed by the site’s automated tests (they need a ROS 2 Jazzy installation or the robot).

ros2 run tf2_ros tf2_echo camera_optical lidar         # the transform in use: converts lidar points into the camera frame
ros2 run tf2_tools view_frames                         # is the tree what you expect?
ros2 topic hz /camera/image_raw
ros2 topic hz /points
ros2 bag record -o align_test /camera/image_raw /camera/camera_info /points /tf /tf_static

Check that the arguments are in the order target source: tf2_echo A B prints the transform that converts points from B into A.

5. Step-by-step investigation

  1. Check the direction first. Pick a physical point 5 m straight ahead of the camera. Compute by hand (or with the Coordinate Transformation Calculator) where it is in the LiDAR frame, then check the live data agrees. A wrong answer means direction or axes, not calibration noise.
  2. Look at the error pattern with the robot still. Use the Camera–LiDAR Projection Visualizer to match what you see: the same shift at every distance suggests a rotation error; a shift shrinking with distance suggests translation; a pattern varying across the image suggests intrinsics.
  3. Measure the translation. Compare the transform’s translation with the measured mounting offsets. A few centimetres is enough to explain near-field misalignment.
  4. Test for time offset. Rotate the robot in place at a steady rate. If alignment is good when still and a constant lateral offset appears while turning, estimate the offset as the lateral shift divided by (focal length × turn rate in rad/s).
  5. Check intrinsics. Compare a straight edge across the image: curved after undistortion means wrong distortion coefficients.
  6. Re-calibrate only after the above. Redo the extrinsic calibration with the sensors rigidly mounted, then repeat the test.

6. Corrective actions

  • Correct the transform direction or the frame convention (invert, or fix the optical-frame rotation).
  • Re-run the extrinsic calibration and store the result with its direction stated in a comment.
  • Fix a time offset at the source (hardware sync, driver stamping) or add a calibrated offset in the pipeline.
  • Re-do the intrinsic calibration if distortion is the cause.
  • Re-mount rigidly and re-calibrate after any mechanical change.

7. Validation after the fix

  • A target at three distances projects within your pixel tolerance, with the robot still.
  • The alignment stays within the same tolerance while turning at the maximum operating rate.
  • The result is repeatable after a power cycle and after a drive over a bump.
  • The calibration, its date and the validation images are saved.

References

  • Module 6 guides (frame conventions, projection, extrinsic calibration): see the module overview.