E · System design challenge 3

Challenge 3: Vision-Guided Manipulation

A camera sees parts on a table; an arm must pick them reliably. Where does the error come from, and how will you prove the cell works?

Uses modules 06, 08, 16
ROS 2 Jazzy

1. Problem statement

A cell has a 6-axis arm, a gripper and a camera, and must pick randomly placed parts from a table and place them in a fixture. The deliverable is an architecture, a calibration and validation plan, and measured evidence that the pick success rate meets a stated requirement.

2. Constraints

Requirement: a stated pick success rate with a confidence level, over the part variety and lighting of the site.
Hand-eye calibration error is smaller than the gripper’s tolerance for the part.
Every grasp is checked after closing and after lifting (width and force).
A collision-free plan with the table, fixture and neighbours in the planning scene.
Cycle time and safety limits (speed near people) are part of the requirement.
The camera, gripper and arm are time-synchronised well enough that a moving part is not mislocated.

3. Architecture

camera ─▶ detection ─▶ pose in camera frame ─▶ T_base_cam (hand-eye) ─▶ grasp, pre-grasp, retreat ─▶ MoveIt 2 (planning scene) ─▶ arm + gripper
                                                    gripper width/force ─▶ grasp validation ─▶ retry or escalate

Item	Decision to make
Gripper	Type, force with safety factor, finger design
Camera	Eye-in-hand or eye-to-hand, resolution, lighting
Calibration	Intrinsics, hand-eye, validation on held-out points
Grasp planning	Approach direction, pre-grasp, retreat, collision scene
Validation	Success rate with an interval, failure classification

4. Implementation stages

Requirements. Parts, tolerances, lighting, cycle time and required success rate with confidence.
Gripper. Choose and size the gripper from the force arithmetic (selecting a gripper).
Camera and lighting. Choose the camera type and mounting; control lighting (see the cameras guide in Module 2).
Calibration. Intrinsics, then hand-eye calibration with held-out validation (camera to robot hand eye calibration).
Grasp poses. Detection to grasp, pre-grasp and retreat (from detection to grasp pose).
Planning. Configure MoveIt 2 and check the description, groups, limits and controllers (configuring moveit 2).
Grasp validation. Classify and recover from failures (grasp validation and failure recovery); see the troubleshooting page.
Time and latency. Measure end-to-end latency from image to motion (measuring end to end latency).
Validation. Run enough trials to demonstrate the requirement (trials and intervals).

5. Validation criteria

Hand-eye validation error on held-out checkpoints is below the requirement at several places in the workspace.
The success rate has a 95 % lower bound above the requirement over the part and lighting variety.
Every failure is classified (missed, wrong, weak, slip, dropped, collision) and the counts reported.
The cell stops safely on a link or camera failure, and the speeds near people match the risk assessment.

6. Engineering trade-offs

Eye-in-hand vs eye-to-hand: a wrist camera sees close detail and moves; a fixed camera has a stable view and a fixed calibration.
Parallel jaw vs vacuum: precision and force against surface sensitivity.
Learned grasp planner vs geometric rules: flexibility against validation effort.
Cycle time vs reliability: slower approach and extra checks raise success and lower throughput.

References

Tools and guides for this challenge are linked in the stages. Sources used by those guides: the MoveIt documentation (hand-eye calibration, pick and place, motion planning, Servo), read 2026-11-06. This challenge is a design task: no performance number in it is a claim about a product.