Module 08 · Phase 3: Sensor integration
Robotics Perception & Sensor Integration
Build and debug an obstacle-detection pipeline from camera and LiDAR: filtering, ground segmentation, clustering, detection, association, tracking and throughput.
This overview is open. 6 of 6 guides are written; the others show Coming soon until they are complete and tested.
Questions this module answers
- How do I build an obstacle detection pipeline?
- How do I detect objects with camera and LiDAR and associate the results?
- How do I diagnose poor point-cloud quality and handle dynamic objects?
- How do I validate detection accuracy and optimize throughput?
What it covers
- Image and point-cloud processing and filtering
- Ground segmentation, clustering
- 2D/3D detection, tracking, semantic segmentation, depth estimation
- Camera–LiDAR fusion
- Sensor placement for perception
- Pipeline optimization
Guides and design decisions
A. Engineering Guides (5)
B. Design & Selection (1)
Diagnostics, tools and challenges
C. Troubleshooting (2)
- LiDAR point cloud is distortedWalls look curved or doubled while the robot moves or turns.
- Camera and LiDAR detections do not alignProjected LiDAR points land beside the object in the image.
E. Design challenges (2)
- Challenge 1: Design an Autonomous Mobile RobotIndoor differential-drive robot: sensors, motor sizing, compute, power, ROS 2, LiDAR+IMU, SLAM, Nav2, tuning and validation.
- Challenge 3: Vision-Guided ManipulationCamera, gripper, calibration, object pose, grasp planning, motion and grasp validation.
Practical deliverables
- Camera–LiDAR perception example
- Point-cloud processing workflow
- Perception debugging guide
Read first: existing FixTheRobot lessons and labs
This module builds on these instead of repeating them.
- Ground and Plane Estimation Perception · Lesson
- Clustering and Object Separation Perception · Lesson
- Precision, Recall and IoU Perception · Lesson
- Early Fusion, Late Fusion and Object-Level Fusion Perception · Lesson
- Process & Transform a Point Cloud Labs · Lab
- Build a LiDAR Obstacle Detection Pipeline Labs · Lab