This lab works on phones, but writing code is much easier on a laptop.
Perception · Sensor Fusion and Scene Understanding · P8-L4 · Advanced
The camera knows exactly which way the pallet is and has no idea how far it is. The lidar knows exactly how far and is a little vague about the direction. Together they know both, if the robot trusts each one where it is good.
You will weigh them by direction, write the fusion of two Gaussians, and refuse to fuse readings that cannot be about the same object.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 explores per-direction weights. Stage 2 writes the information-form fusion with 2 × 2 covariances. Stage 3 adds a chi-square consistency test that rejects conflicting observations. All observations are simulated with stated covariances. Your code is graded on hidden data; the true positions are kept by the grader.
Educational simplification: 2D positions, independent Gaussian errors with exactly known covariances, two sensors and known correspondence apart from the injected conflicts. In practice the covariances are estimated and often wrong, errors are correlated between sensors that share a platform, and association itself fails. Fusion is only valid after calibration, synchronisation and frame alignment (Lab P8-L3). All data in this lab are simulated.
Get notified when new robotics labs are released. One email per new lab, nothing else.