Perception · Theory · P9
P9-T4 — Perception for Manipulation and Autonomous Navigation
The same camera serves two very different customers. One needs centimetres at arm’s length, the other metres at a hundred metres, and both need to know when the answer is not to be trusted.
Perception is not an end in itself; its output is consumed by a robot task, and tasks differ in what they need. A manipulator needs the pose of one object to a few millimetres, once, at close range, and tolerates a second to look. A mobile robot needs free space and moving obstacles at long range, continuously, to within a fraction of its own width, with a latency measured in tens of milliseconds. This lesson compares the two sets of requirements, turns accuracy and tolerance into probabilities of success, and shows where perception meets learned policies. It prepares the capstone, whose report is exactly the kind of interface a navigation task consumes. For how learned policies take perception as their input, see the VLA track and the state, observation and belief lesson of the Physical AI track.
Learning objectives
After this lesson, you should be able to:
- Compare the perception requirements of manipulation and navigation in accuracy, range, rate and latency.
- Compute the probability that a pose error stays within a grasp tolerance.
- Compute clearance requirements for a navigating robot given localisation and obstacle position errors.
- Explain how perception outputs are used as observations by learned policies and what that demands.
The idea: requirements come from the task
Ask what the consumer needs, and the design follows. Two tasks will make the contrast.
| Manipulation (grasp a part) | Navigation (drive through a building) | |
|---|---|---|
| What is needed | the pose (position and orientation) of one object, its shape, free space around the gripper | free space, static map, moving obstacles, the robot’s own position |
| Range | arm’s length: 0.3 to 1 m | metres to tens of metres |
| Accuracy | millimetres to a centimetre | a fraction of the robot’s width, 5 to 20 cm |
| Rate and latency | a few frames before the grasp; can stop and look again | continuous at 10 to 30 Hz, tens of milliseconds |
| Typical sensors | wrist RGB-D camera, tactile sensing | lidar, cameras, radar, odometry |
| Characteristic failure | wrong pose from occlusion by the gripper, reflective parts, clutter | missed obstacle, localisation drift, stale map, mistaken free space |
| What can be done on doubt | move the camera, re-grasp, ask | slow down, stop, replan |
Manipulation
The robot must know where the object is relative to its gripper, in the robot’s base frame (Lesson P8-T3), and its orientation, so the pose has six numbers (3D position, 3D rotation). The camera is usually on the wrist or fixed above the work area, so the extrinsic calibration between camera and arm (hand-eye calibration) matters as much as the detector. Accuracy needs are set by the gripper tolerance, as the figure shows, and by contact: a small error in free space is harmless, a small error at the moment of contact is a collision.
Navigation
The robot must know where it can go and what is moving. The consumer is a planner that uses a map of free space and a list of dynamic objects with velocity (Lesson P8-T6). Errors are measured against the robot’s radius: an obstacle placed 0.2 m wrong matters only if the margin is smaller than that. The robot’s own position error adds to the obstacle’s position error, and the stopping distance (Lab P9-L1) sets how far ahead perception must work.
Learned policies as consumers
Modern systems often feed perception output, or raw images, into a learned policy that outputs actions, as in the VLA track. The policy then plays the role of the planner or controller. Three consequences follow. The policy learned from the observations of its training set, so the perception’s errors at run time must look like those in training; a calibration shift or a new camera is a distribution shift (Lesson P6-T6). The policy has no mechanism for saying “my perception is unreliable” unless one is provided, so the pipeline’s monitoring and confidence outputs need to reach it. And the interface (frames, rates, timestamps) must be exactly what it was trained with.
The mathematics: tolerance, clearance and error budgets
Success probability of a grasp. Suppose the lateral error of the estimated object position is Gaussian with standard deviation σ and the gripper tolerates ±a. The probability that the error is within the tolerance is
- a
- the tolerance: half the gap between the open fingers and the object, in the same unit as σ.
- σ
- the standard deviation of the position error along that axis.
- erf
- the error function: erf(1/√2) = 0.683, erf(2/√2) = 0.954, erf(3/√2) = 0.997; that is, ±1σ, ±2σ and ±3σ.
If several independent axes each need to be within tolerance, the probabilities multiply. A 2 × tolerance relative to σ (a = 2σ) gives 95.4% per axis.
Clearance for navigation. A robot of radius R passing an obstacle needs a distance to its centre line of at least
- R
- the robot radius (including any overhang).
- m
- a fixed safety margin.
- σobs, σloc
- the standard deviations of the obstacle’s position and of the robot’s own position, in the direction of the clearance (they add as variances).
- k
- a multiplier: 2 for about 95%, 3 for 99.7% of the Gaussian errors.
Perception range. The robot must see an obstacle at least a stopping distance away (Lab P9-L1): dsee ≥ v L + v²/(2a) + margin, with L the total latency.
Worked examples: grasping and passing
Grasp tolerance. A gripper opens to 8 cm for a 6 cm object: the tolerance is a = (8 − 6)/2 = 1 cm on each side.
| σ of the position error | a/σ | P(|e| ≤ 1 cm) = erf(a/(σ√2)) |
|---|---|---|
| 0.5 cm | 2 | 0.954 |
| 1 cm | 1 | 0.683 |
| 2 cm | 0.5 | 0.383 |
A depth camera with σ = 1 cm fails to align one grasp in three, from lateral error alone; a second axis with the same σ makes the success rate 0.683² = 0.47. Improving the pose estimate from 1 cm to 0.5 cm (a closer camera, a better calibration, averaging over frames) raises per-axis success to 95.4% and two-axis success to 91%. Alternatively, a wider opening or a compliant gripper raises a instead. The better of the two is the cheaper change.
Passing an obstacle. A robot of radius 0.35 m with a margin of 0.1 m. The obstacle position error is σobs = 0.15 m and the robot’s localisation error σloc = 0.10 m. With k = 3:
A corridor 2.0 m wide, with the obstacle in the middle, leaves 1.0 m to either side of the obstacle’s centre: just enough. If the obstacle estimate comes from a misaligned camera (0.5 m off at range, Lab P8-L3) the true clearance may be as little as 0.5 m, and the robot clips the obstacle although the plan said 1.0 m. The clearance formula covers random error, not a calibration fault. That needs a separate check.
Connecting to the capstone. The report built in Lab P9-L2 gives obstacles in the robot frame with range and bearing, and Stage 4 turns it into a stop-or-go decision using the latency, the speed and the braking. It is a navigation consumer. A manipulation consumer would need a pose with orientation and a wrist-camera calibration, which the same ingredients (lifting, extrinsics, filters) provide at a different accuracy.
Engineering implications and limitations
Write down the consumer’s requirements. Accuracy per axis, range, rate, latency, frame and what it does on a doubt. Test the perception against these numbers, not against a generic metric.
Calibrate for the task. Hand-eye calibration for manipulation, camera-to-robot extrinsics for navigation, each verified against a measured target (Lesson P8-T3).
Use the right sensor geometry. A wrist camera close to the part gives better accuracy than a fixed camera far away (the depth error grows with range, Lesson P8-T2); a second view removes depth ambiguity and occlusion by the gripper.
Look again. For manipulation, moving the camera and averaging several estimates reduces random error by about √n. Plan for a verification view before contact.
Give the consumer the uncertainty. A planner can widen margins when σ is large; a grasp planner can pick a more tolerant grasp. A point estimate hides what they need.
Feed learned policies what they were trained on. Keep the observation format, frames and rates identical to training, and monitor for shift in the inputs. Provide the policy, or its supervisor, with the perception’s confidence and conflict flags.
Test the whole chain. Run sensor to action on logged scenes with injected faults (Lesson P9-T6) and check that the outcome is safe: successful grasps or a controlled stop.
Common misconceptions
“Good perception means a high detection score.”
Good means meeting the consumer’s requirement: millimetres for a grasp, a stopping distance for navigation. A perfect classification with a 2 cm pose error fails the grasp.
“Manipulation and navigation need the same sensor setup.”
They differ in range, accuracy, rate and failure handling. A wrist camera that is ideal for grasping is the wrong tool for obstacle avoidance at 10 m.
“Averaging more frames always fixes the error.”
It reduces random error as 1/√n; a calibration bias or a systematic occlusion is not reduced at all.
“A learned policy handles perception errors by itself.”
It handles those it saw in training. Unfamiliar errors, such as a camera moved on its mount, are shifts it cannot detect without help.
Engineering takeaways
- Requirements come from the consumer: manipulation needs mm to cm pose accuracy at arm’s length, navigation needs free space and moving obstacles at range, with latency in tens of milliseconds.
- Grasp success per axis is erf(a/(σ√2)): with a = 1 cm, σ = 0.5 cm gives 95.4%, σ = 1 cm 68.3%, σ = 2 cm 38.3%; axes multiply.
- Navigation clearance d = R + m + k·√(σobs² + σloc²) covers random error (0.99 m in the example), not a calibration fault.
- Look again and average for random error (1/√n); fix bias by calibration.
- Learned policies consume perception as observations: match the training interface, monitor input shift and pass confidence on.
References and further reading
A benchmark set, a benchmark paper and textbooks. The grip tolerance, error values and clearances in the examples are illustrative.
- Calli, B. et al. (2015). The YCB Object and Model Set: Towards Common Benchmarks for Manipulation Research. ICAR / IEEE Robotics and Automation Magazine.
- Hodaň, T. et al. (2018). BOP: Benchmark for 6D Object Pose Estimation. ECCV.
- Siciliano, B. and Khatib, O. (eds.) (2016). Springer Handbook of Robotics, 2nd ed.: perception for manipulation and mobile robots.
- Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: localisation and mapping for navigation.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A parallel gripper opens to 5 cm around a 4 cm part. What is the tolerance, and what is P(|e| ≤ tolerance) for σ = 0.25 cm?
Explanation
a = 0.5 cm; a/σ = 2, so erf(2/√2) = 0.954, about 95%.
Question 2
A robot of radius 0.4 m uses a margin of 0.1 m and k = 2. Obstacle error σobs = 0.2 m, localisation σloc = 0.1 m. What clearance is needed?
Explanation
d = 0.4 + 0.1 + 2 × sqrt(0.04 + 0.01) = 0.5 + 2 × 0.2236 = 0.947 m.
Question 3
A grasp succeeds if both the x and y errors are within ±1 cm, each with σ = 0.7 cm. Roughly what is the success rate?
Explanation
Per axis, a/σ = 1.43, erf(1.43/√2) ≈ 0.847. Two axes: 0.847² ≈ 0.72, about 72%.
What’s next
Next: running perception on a real robot: real-time constraints, latency tails and deployment on limited hardware.
Related content
Nothing below is required to finish this lesson.
- BeforeP9-T3: Open-Vocabulary Detection and Vision-Language Models
Explain how language and vision are linked and what open-vocabulary detection can and cannot do.
- LabP9-L2: Capstone: Build an End-to-End Perception Pipeline
From simulated camera and range observations to a structured obstacle report for a downstream task, with diagnosis of injected failures.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.