Engineering Guides · Module 16

From detection to grasp pose

A grasp is a pose plus an approach and a retreat. Compute all three in the robot frame from one calibrated detection.

Intermediate
Module 16: Manipulation & Robotic Arm Integration
Written for ROS 2 Jazzy where ROS 2 is used
Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

A detector gives an object position in the camera frame. The arm needs a gripper pose in its own frame, an approach path that does not hit the object or the table, and a retreat after the object is held.

2. When is this needed?

Building a pick pipeline.
When approaches collide with the object.
Adding a new object type.

3. Selection and design decisions

The MoveIt pick and place tutorial (read 2026-11-06) describes the moveit_msgs/Grasp message with a grasp_pose (“pose of the end effector in which it should attempt grasping”), a pre_grasp_approach (the direction from which to approach and the distance to travel), a post_grasp_retreat (the direction in which to move once the object is grasped and the distance), and the end-effector posture before and during grasping. The transform from the camera to the robot base comes from the hand-eye calibration.

Overview
Step	Computation
Object in robot frame	p_base = T_base_cam · p_cam
Grasp pose	Object centre plus an offset along the approach, with the gripper orientation from the object’s principal axes or a grasp planner
Pre-grasp	Grasp pose backed off along the approach direction by the approach distance
Retreat	Grasp pose moved along the retreat direction by the retreat distance
Collision check	Check each pose and the path against the planning scene, including the table and the object

4. Advantages and disadvantages

Top grasps are simple and fail for objects too wide or low; side grasps need more clearance.
Learned grasp planners handle unknown objects and need data and validation.
Longer approaches are safer and slower.

5. Requirements and constraints

The calibration error is smaller than the gripper’s tolerance for the object.
Approach and retreat paths are collision-free.
Orientation limits and joint limits are respected.

6. Architecture

detection (camera frame) ─▶ T_base_cam ─▶ object pose in base ─▶ grasp, pre-grasp, retreat ─▶ planner (collision-checked) ─▶ execute

7. Implementation

Transform the detection into the robot frame.
Choose the grasp orientation and the approach direction.
Compute the pre-grasp and retreat poses (below).
Send them to the planner with the planning scene including the table and the object.
Check the result on the real robot at low speed first.

import numpy as np
def T(R, t):
    M = np.eye(4); M[:3, :3] = R; M[:3, 3] = t; return M
def rotz(d):
    a = np.radians(d); c, s = np.cos(a), np.sin(a); return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1.0]])
T_base_cam = T(np.diag([1.0, -1.0, -1.0]), [0.6, 0.0, 0.8])                                      # camera 0.8 m above the table origin, optical axis pointing down
p_cam = np.array([0.05, -0.02, 0.60])                                                           # object centre in the camera frame
p_base = (T_base_cam @ np.append(p_cam, 1.0))[:3]
approach = np.array([0.0, 0.0, -1.0])                                                           # approach straight down in the base frame
grasp = p_base + np.array([0.0, 0.0, 0.01])                                                     # grasp 1 cm above the object centre for a top grasp
pre = grasp - approach * 0.10                                                                   # pre-grasp: 10 cm back along the approach direction
retreat = grasp + np.array([0.0, 0.0, 0.15])                                                    # post-grasp retreat 15 cm up
for n, p in (("object in base frame", p_base), ("pre-grasp", pre), ("grasp", grasp), ("retreat", retreat)):
    print(f"{n:22s}", np.round(p, 3))
print("approach distance:", round(float(np.linalg.norm(grasp - pre)), 3), "m")

Printed (checked by the site’s tests):

object in base frame   [0.65 0.02 0.2 ]
pre-grasp              [0.65 0.02 0.31]
grasp                  [0.65 0.02 0.21]
retreat                [0.65 0.02 0.36]
approach distance: 0.1 m

Reading it: an object seen 0.60 m in front of a downward-looking camera mounted 0.8 m above the table lands at (0.65, 0.02, 0.20) m in the base frame; the grasp is 1 cm above its centre, the pre-grasp is 10 cm back along the downward approach (0.31 m high) and the retreat is 15 cm above the grasp. The setup and numbers are illustrative.

8. Real examples

Box picking: top grasp with a 10 cm approach and a 15 cm retreat.

9. Failure scenarios

Using the camera frame orientation convention for the gripper without conversion.
An approach that crosses another object.
No retreat clearance, so the object is dragged.

10. Troubleshooting

Overlay the grasp poses in RViz.
Check the frame conventions and the sign of the approach.
Check the planning scene contains the neighbours.

11. Validation

The example prints the lines shown (executed by the site tests).
The arm reaches the pre-grasp, grasp and retreat without collision in 20 trials.

13. Further references

MoveIt pick and place tutorial: moveit.picknik.ai, read 2026-11-06 (moveit_msgs/Grasp: grasp_pose, pre_grasp_approach, post_grasp_retreat).
MoveIt hand-eye calibration tutorial: moveit.picknik.ai, read 2026-11-06 (the tutorial text still contains ROS 1-era build commands; its calibration procedure is what is used here).