This lab works on phones, but writing code is much easier on a laptop.
Perception · Sensor Fusion and Scene Understanding · P8-L1 · Intermediate
The detector says there is a person in a box at pixels (60, 30)–(75, 70). The planner needs to know how far away the person is, and on which side.
You will take a robust distance from the depth image, lift the box to a 3D point and report it in the robot’s own frame.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 compares ways of choosing a depth for a box. Stage 2 writes the lifting with the pinhole model. Stage 3 converts to the robot frame and reports range and bearing. All images are simulated, and the boxes come from a simulated detector. Your code is graded on hidden scenes against the true object positions, which the grader keeps.
Educational simplification: upright flat-fronted objects on a flat ground plane, a pinhole camera looking horizontally with exactly known intrinsics, a depth image registered to the colour image, depth noise growing as the square of the distance and 25% random holes. Real depth sensors also have systematic errors, edge artefacts and registration errors. All data in this lab are simulated.
Get notified when new robotics labs are released. One email per new lab, nothing else.