Module 06 · Phase 3: Sensor integration · Flagship
Camera–LiDAR Calibration & Coordinate Transformations
Get frames and extrinsics right: PL = TLC × PC with the direction of every transform spelled out, camera projection, calibration validation and TF debugging.
This overview is open. 7 of 7 guides are written; the others show Coming soon until they are complete and tested.
Questions this module answers
- How do I transform a point from camera to LiDAR frame and back?
- How do I project a 3D point into a 2D image, and how is that different from estimating depth from a detection?
- How do I associate an image detection with a 3D LiDAR cluster?
- How do I validate extrinsics and diagnose misaligned detections?
What it covers
- Frame conventions: camera, LiDAR, IMU, base, odom, world
- Rotation matrices, Euler angles, quaternions, homogeneous matrices
- Intrinsic and extrinsic calibration; camera–LiDAR, LiDAR–IMU, camera–IMU
- Online extrinsic calibration
- Python, C++ and ROS 2 TF2 examples
Guides and design decisions
A. Engineering Guides (7)
- Frame conventions and transform direction
- Transforming points between camera, LiDAR and base frames
- Projecting LiDAR points into the image
- Camera–LiDAR extrinsic calibration and validation
- LiDAR–IMU and camera–IMU calibration
- TF2 in ROS 2: static transforms and debugging
- Associating detections with LiDAR clusters
Diagnostics, tools and challenges
C. Troubleshooting (3)
- Incorrect TF transformsThe model or point cloud appears rotated, mirrored or offset in RViz.
- Extrinsic calibration is wrongFused data disagrees consistently by a fixed rotation or offset.
- Camera and LiDAR detections do not alignProjected LiDAR points land beside the object in the image.
D. Interactive tools (2)
- Coordinate Transformation CalculatorXYZ, rotation and translation in; transformed point, homogeneous matrix and intermediate steps out.
- Camera–LiDAR Projection VisualizerProject 3D points into an image, change the extrinsics and see the reprojection error.
E. Design challenges (2)
- Challenge 2: Outdoor-to-Indoor Autonomous NavigationGNSS outdoors to LiDAR-inertial or visual-inertial indoors: sensor and frame architecture, sync, calibration, estimation, handover and recovery.
- Challenge 3: Vision-Guided ManipulationCamera, gripper, calibration, object pose, grasp planning, motion and grasp validation.
Practical deliverables
- Coordinate transformation calculator
- Camera–LiDAR projection visualizer
- TF debugging checklist
Read first: existing FixTheRobot lessons and labs
This module builds on these instead of repeating them.
- Coordinate Frames and Sensor Geometry Perception · Lesson
- Camera Models and Pinhole Projection Perception · Lesson
- Intrinsic and Extrinsic Calibration Perception · Lesson
- Projection and Reprojection Errors Perception · Lesson
- Robot Poses and Coordinate Frames SLAM · Lesson
- TF Frames and Coordinate Consistency SLAM · Lesson
- Transform Points Between Coordinate Frames SLAM · Lab
- Fix Incorrect Coordinate Frames SLAM · Lab