FixTheRobot Explore Robotics / Perception

Learning track

Perception

How a robot turns raw sensor measurements into objects, geometry, identities and a scene it can act on. Theory lessons for the ideas, labs where you diagnose and repair real failures.

54 of 54 theory lessons and 29 of 29 labs are open. Items marked coming soon are not yet available. Start with P1-T1: What Robot Perception Is.

What perception is

Perception is the step between measuring and deciding: it extracts useful information from raw sensor data. The central pipeline of this track is shown below; every module teaches one or two of its stages and the capstone assembles all of them.

The perception pipelineSix stages in a row: raw sensor data, preprocessing, features and representations, detection and estimation, tracking and scene understanding, and robot decision-making.Raw datacamera · LiDAR · depthPreprocessfilter · calibrateFeaturesedges · points · clustersDetectobjects · geometryTrackidentity · motionDecidestop · grasp · steer
The perception pipeline: from raw data to a decision. Each stage can fail on its own, and each lab breaks one of them.

Perception and its neighbours

AreaQuestion it answersWhere it is taught
SensingWhat does the device measure?Sensors track: raw measurements, point clouds, preprocessing
PerceptionWhat is in the measurements: objects, surfaces, features, identities, distances?This track
LocalizationWhere is the robot?Localization & SLAM
MappingWhat does the environment look like, as a persistent representation?Localization & SLAM
PlanningWhat should the robot do, given goals and constraints?Navigation (coming soon)

The areas overlap, for example visual odometry uses features from Module 4 and SLAM uses object detections as landmarks, but each is taught once and linked from the others. Related: Understand & Visualize a Point Cloud, Process & Transform a Point Cloud and Build a LiDAR Obstacle Detection Pipeline (the Sensors labs that Module 5 builds on); the Perception research area.

How the track works

Every module pairs theory with practice. A lesson explains an idea with intuition, the mathematics, a worked robotics example and its limits. The lab that follows hands you a perception system that fails in an observable way and you follow the loop Problem → Experiment → Diagnose → Fix → Evaluate → Understand. Labs run in your browser in Python and are graded on hidden scenarios; the simulator keeps its ground truth to itself. All data are simulated and each lab says what is simplified. Where a lab shows detector outputs, they are labelled as simulated predictions, not the output of a trained network.

Not sure where to begin? Start with Module 1; if you already know images and geometry, jump to the module you need using the list below.

Module 1

Perception Foundations

What perception is, how a measurement becomes a representation, the pipeline, sensor geometry, uncertainty and how to judge a perception system.

You will be able to: separate sensing from perception, localization, mapping and planning; describe the perception pipeline and name where each failure enters; express measurements in the right coordinate frame; explain noise, occlusion and ambiguity and evaluate a system with the right metric.

All 9 items available

Theory (6)

  1. P1-T1What Robot Perception IsTheory
  2. P1-T2Sensors, Measurements, Representations and SemanticsTheory
  3. P1-T3The Perception PipelineTheory
  4. P1-T4Coordinate Frames and Sensor GeometryTheory
  5. P1-T5Noise, Uncertainty, Occlusion and AmbiguityTheory
  6. P1-T6How to Evaluate a Perception SystemTheory

Labs (3)

  1. P1-L1Inspect and Interpret Sensor ObservationsBrowser lab
  2. P1-L2Find the Misleading MeasurementsBrowser lab
  3. P1-L3Build a First Perception PipelineBrowser lab

Module 2

Image Processing for Robotics

Pixels, colour spaces, filtering, thresholding, segmentation, edges and contours, and why these pipelines fail under real conditions.

You will be able to: represent an image as numbers and convert between RGB, grayscale and HSV; filter noise and choose between filters; segment objects with thresholds and connected regions; extract edges and contours and diagnose preprocessing failures.

All 9 items available

Theory (6)

  1. P2-T1Pixels, Image Dimensions and Color RepresentationsTheory
  2. P2-T2RGB, Grayscale and HSVTheory
  3. P2-T3Filtering and Noise ReductionTheory
  4. P2-T4Thresholding and SegmentationTheory
  5. P2-T5Edges, Contours and Simple Geometric FeaturesTheory
  6. P2-T6Image Preprocessing FailuresTheory

Labs (3)

  1. P2-L1Detect Colored Objects in an ImageBrowser lab
  2. P2-L2Tune Thresholds Under Changing LightingBrowser lab
  3. P2-L3Diagnose a Failing Image-Processing PipelineBrowser lab

Module 3

Geometry and Camera Perception

The pinhole camera, calibration, coordinates, perspective and depth ambiguity, depth cameras and reprojection error.

You will be able to: project 3D points into an image and back-project a pixel to a ray; explain intrinsic and extrinsic parameters and the effect of getting them wrong; recover simple geometry from depth; measure and interpret reprojection error.

All 9 items available

Theory (6)

  1. P3-T1Camera Models and Pinhole ProjectionTheory
  2. P3-T2Intrinsic and Extrinsic CalibrationTheory
  3. P3-T3Image Coordinates Versus Camera CoordinatesTheory
  4. P3-T4Perspective and Depth AmbiguityTheory
  5. P3-T5Depth Cameras and RGB-D DataTheory
  6. P3-T6Projection and Reprojection ErrorsTheory

Labs (3)

  1. P3-L1Project 3D Points into an ImageBrowser lab
  2. P3-L2Investigate the Effect of Camera ParametersBrowser lab
  3. P3-L3Recover Geometry from Depth ObservationsBrowser lab

Module 4

Classical Feature Detection and Matching

Corners and local features, descriptors, correspondences, robust estimation with RANSAC and geometric consistency, and failure cases.

You will be able to: detect corners and describe local patches; match features and measure the matches; reject outliers with RANSAC and geometric consistency; predict when repetition or lack of texture breaks matching.

All 9 items available

Theory (6)

  1. P4-T1Corners, Edges and Local FeaturesTheory
  2. P4-T2Feature Descriptors and MatchingTheory
  3. P4-T3Correspondences and OutliersTheory
  4. P4-T4Robust Estimation and RANSACTheory
  5. P4-T5Geometric ConsistencyTheory
  6. P4-T6Repetitive and Textureless ScenesTheory

Labs (3)

  1. P4-L1Detect Image FeaturesBrowser lab
  2. P4-L2Match Features Between ImagesBrowser lab
  3. P4-L3Reject Incorrect Correspondences and Measure the ImprovementBrowser lab

Module 5

3D Perception and Point Clouds

Point-cloud representations, local geometry, plane estimation, clustering, bounding boxes and the effect of occlusion and sparsity, building on the Sensors labs.

You will be able to: describe a point cloud and its neighbourhood structure; estimate local geometry and a ground plane robustly; cluster and box objects and estimate their size and distance; diagnose failures caused by sparsity and occlusion.

Builds on existing labs: Understand & Visualize a Point Cloud (Sensors), Process & Transform a Point Cloud (Sensors), Build a LiDAR Obstacle Detection Pipeline (Sensors).

All 9 items available

Theory (6)

  1. P5-T13D Point-Cloud RepresentationsTheory
  2. P5-T2Neighborhoods and Geometric FeaturesTheory
  3. P5-T3Ground and Plane EstimationTheory
  4. P5-T4Clustering and Object SeparationTheory
  5. P5-T5Bounding Boxes and Geometric Object PropertiesTheory
  6. P5-T6Occlusion, Sparse Measurements and Density ChangesTheory

Labs (3)

  1. P5-L1Estimate Surface Normals and PlanarityBrowser lab
  2. P5-L2Fit a Ground Plane RobustlyBrowser lab
  3. P5-L3Debug a Failed 3D Detection PipelineBrowser lab

Module 6

Object Detection and Semantic Understanding

Classification, detection and segmentation, bounding boxes and confidence, training data, evaluation with IoU, precision and recall, and failure under domain shift.

You will be able to: distinguish classification, detection and segmentation outputs; compute IoU, precision, recall and average precision; choose a confidence threshold for an application; explain false positives, false negatives and domain shift.

All 10 items available

Theory (6)

  1. P6-T1Classification, Detection and SegmentationTheory
  2. P6-T2Bounding Boxes and Confidence ScoresTheory
  3. P6-T3Semantic and Instance SegmentationTheory
  4. P6-T4Training Data and AnnotationTheory
  5. P6-T5Precision, Recall and IoUTheory
  6. P6-T6False Positives, False Negatives and Domain ShiftTheory

Labs (4)

  1. P6-L1Interpret Detection OutputsBrowser lab
  2. P6-L2Evaluate Predictions Against Ground TruthBrowser lab
  3. P6-L3Investigate Confidence Thresholds and Missed DetectionsBrowser lab
  4. P6-L4Compare Performance Across Scene ConditionsBrowser lab

Module 7

Multi-Object Tracking and Temporal Perception

Detections across frames, association, track state and motion estimation, Kalman filter intuition, track lifecycle, occlusion and identity switches.

You will be able to: distinguish a detection (an observation) from a track (an estimate over time); associate detections to tracks with gating and assignment; estimate track motion with a simple filter; manage track birth, persistence and death through occlusion.

All 10 items available

Theory (6)

  1. P7-T1Detection Across Consecutive FramesTheory
  2. P7-T2Object Association and IdentityTheory
  3. P7-T3Track State and Motion EstimationTheory
  4. P7-T4Simple Filtering and Kalman Filter IntuitionTheory
  5. P7-T5Track Initiation, Persistence and DeletionTheory
  6. P7-T6Occlusion, Identity Switches and Missed DetectionsTheory

Labs (4)

  1. P7-L1Track Moving Objects Across Simulated FramesBrowser lab
  2. P7-L2Repair an Object-Association AlgorithmBrowser lab
  3. P7-L3Investigate How Noisy Detections Affect Track QualityBrowser lab
  4. P7-L4Recover from Temporary OcclusionBrowser lab

Module 8

Sensor Fusion and Scene Understanding

Why robots combine sensors, camera and LiDAR strengths and weaknesses, alignment and timing, early, late and object-level fusion, uncertainty and scene representations.

You will be able to: explain what each sensor contributes and what it cannot; align sensors in space and time; choose a fusion architecture; combine estimates with uncertainty and detect conflicts.

All 10 items available

Theory (6)

  1. P8-T1Why Robots Combine Multiple SensorsTheory
  2. P8-T2Complementary Strengths and Weaknesses of Cameras and LiDARTheory
  3. P8-T3Coordinate Alignment and TimestampsTheory
  4. P8-T4Early Fusion, Late Fusion and Object-Level FusionTheory
  5. P8-T5Uncertainty and Conflicting ObservationsTheory
  6. P8-T6Scene Representations for Downstream Robotics TasksTheory

Labs (4)

  1. P8-L1Combine Camera Detections with Depth InformationBrowser lab
  2. P8-L2Compare Camera-Only and Depth-Assisted Distance EstimatesBrowser lab
  3. P8-L3Diagnose Incorrect Coordinate TransformsBrowser lab
  4. P8-L4Fuse Simulated Observations with UncertaintyBrowser lab

Module 9

Modern Robotics Perception and Capstone

CNNs and transformers, foundation and open-vocabulary models, perception for manipulation and navigation, real-time constraints, robustness, and an end-to-end capstone pipeline.

You will be able to: explain what CNNs, transformers and foundation models add to perception; relate perception to manipulation, navigation and vision-language-action systems; budget latency and analyse robustness and failure; build and evaluate an end-to-end perception pipeline.

All 8 items available

Theory (6)

  1. P9-T1CNNs and Modern Visual PerceptionTheory
  2. P9-T2Vision Transformers and Foundation ModelsTheory
  3. P9-T3Open-Vocabulary Detection and Vision-Language ModelsTheory
  4. P9-T4Perception for Manipulation and Autonomous NavigationTheory
  5. P9-T5Real-Time Constraints and DeploymentTheory
  6. P9-T6Robustness, Evaluation and Failure AnalysisTheory

Labs (2)

  1. P9-L1Evaluate Robustness and Latency BudgetsBrowser lab
  2. P9-L2Capstone: Build an End-to-End Perception PipelineCapstone lab