FixTheRobot Explore Robotics / Perception / P9 / P9-L2

This lab works on phones, but writing code is much easier on a laptop.

Perception · Modern Robotics Perception and Capstone · P9-L2 · Advanced

Capstone: Build an End-to-End Perception Pipeline

The robot has a camera, a depth image and a detector. The planner has a question: is anything in my way, close enough that I must stop?

You will build the chain that answers it, find out what breaks it, and use the answer to decide.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().

    Hints

    What this lab teaches

    This capstone joins the modules of the track. Stage 1 configures a pipeline against injected failures. Stage 2 writes it: robust depth, lifting, filtering, extrinsics and the robot-frame report. Stage 3 diagnoses calibration and timing faults from how residuals depend on range and speed. Stage 4 turns the report, with latency and braking, into a stop-or-go decision. The scene, the depth and the detections are simulated, and the detector is a simulator with a stated error model, not a trained network. Your code is graded on hidden scenes whose true positions and faults the grader keeps.

    Theory for this lab

    Modern Robotics Perception and Capstone labs

    1. P9-L1: Evaluate Robustness and Latency Budgets
    2. P9-L2: Capstone: Build an End-to-End Perception Pipeline: this lab.

    Educational simplification: upright flat-fronted objects, a pinhole camera with known intrinsics and a stored extrinsic yaw, depth noise growing as z², 25% holes, simulated detections with false alarms on the ground, static scenes in Stages 1 and 2, and a corridor-and-stopping-distance decision in Stage 4. Real systems add moving objects, tracking, several sensors, uncertainty propagation and many more failure modes. All data in this lab are simulated.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.