M9-T1 — Learning to Grasp

Learning objectives

After this lesson, you should be able to:

• Write grasp success prediction as a classification problem with features measured before the lift, and name what the label does and does not tell you.
• Compare an honest learned predictor, a hand force-balance rule and a leaked-feature model on held-out grasps, and explain the 0.823, 0.833 and 0.9997 accuracies of this lesson.
• Explain why a feature that is available in simulation, or only after the grasp, makes a model look perfect and then fail on hardware.
• Say what a grasp-quality metric, a success predictor and a grasp policy each need from the data and the sensor, and where each can fail.

The intuition

Picture Scout’s gripper above a medicine box. Before closing it knows how far the fingers are from the box’s centre (the offset), how tilted the approach is, an estimate of how grippy the surfaces are, and the squeezing force it will apply. It does not know the box’s mass, and the friction estimate is noisy. After the lift, a slip sensor reports how fast the box slid in the fingers. If the box held, the slip is tiny; if the box dropped, it is large. A model with access to that last number will look wonderful in the lab, and be useless in the one situation that matters, because the number only exists once the answer is known.

The figure is the whole lesson in five bars. A model that always says “holds” is right 59.6% of the time, since about 58% to 60% of these grasps succeed. Four honest features take a logistic regression to 0.823, about the same as a hand rule that assumes a typical 0.8 kg object (0.833). Adding the slip column pushes the same classifier to 0.9997. Take the column away at deployment, which is what a real robot has to do, and the accuracy falls back to 0.596: the model sees “no slip” and says “holds” for everything.

Two different gaps are on display. The gap between 0.823 and 0.9997 is a leak: information the model gets that the robot will not have. The gap between 0.823 and 1.0 is missing information: the object’s mass and true friction, which no feature in the table provides. Only the second is a real property of the task, and it limits any predictor, however large.

The mathematics

The toy grasp and its label

Each row is one attempt. The generator draws an object mass m between 0.3 and 1.3 kg, an offset of the grasp point from the centre of mass of up to 4 cm, a tilt of the gripper up to 30 degrees, a true friction coefficient μ between 0.3 and 1.1 and a squeezing force F between 5 and 25 N. The object is held when two conditions are met: the friction can carry the weight with a safety factor, and the friction torque can resist the torque of the weight about the grasp point.

The factor 2 counts two fingers, 1.3 is a safety factor and 0.02 m is a stand-in for the contact lever arm. These are my modelling choices; real grasps involve contact geometry, pad compliance and dynamics that this toy leaves out. The learner never sees m or the true μ. Its columns are the offset in cm, the tilt in degrees, a measured friction estimate (true μ with 10% Gaussian noise) and the force F. A fifth column, slip in mm/s, is measured after the lift: about 0.5 for a successful grasp and between 3 and 7 for a failed one.

Logistic regression for a yes/no label

With standardised features z (lesson M2-T2) the model is a probability, trained by minimising the cross-entropy loss (lessons M2-T1 and M2-T3):

I trained by batch gradient descent, 400 epochs, step 0.5, no regularisation, with standardisation computed on the training grasps only. The decision rule is p > 0.5. A quality number such as accuracy is only meaningful against the base rate and on grasps the model has not seen (lesson M2-T6).

offset, tilt

Distance of the grasp point from the centre of mass (cm) and gripper tilt (degrees).

μ, μ̂

True friction coefficient (hidden) and its noisy measurement (a feature).

F

Squeezing force in newtons, chosen by the robot before the grasp.

m

Object mass in kilograms; hidden. The generator draws it from 0.3 to 1.3.

slip

Slip speed in mm/s measured after the lift. A leaked feature, not usable before the grasp.

accuracy

Fraction of held-out grasps classified correctly; compare with the base rate 0.596.

Three kinds of tool for the same problem

Classic analytic metrics, such as the force-closure based quality measures of Ferrari and Canny (1992), score a grasp from an object model and a friction assumption. The learned predictors of the 2010s, such as the Dex-Net family (Mahler et al., 2017) and the large-scale self-supervised grasping of Pinto and Gupta (2016) and Levine and colleagues (2018), replace or combine the metric with a model trained on grasp outcomes. Mahler et al. trained on synthetic grasps labelled by an analytic metric and deployed on a real arm, which is exactly the setting in which one must check that the training features exist at deployment time. I describe these systems in one sentence each and have not reproduced their numbers.

A worked example

Every number is from running Node against RLCore.graspData; the seeds are SLAMCore.rng(1..20). For each seed I draw 300 training grasps and then 1,000 test grasps from the same stream, standardise on the training set, train as above and score on the test set. Intervals are 95% Student t over the 20 seeds.

By hand: one grasp, two ways to be wrong

The first three grasps of seed 1, with the hidden values that the learner does not see. Offsets are converted to metres for the torque (offset cm / 100).

The needed force for the guessed mass is 1.3 × 0.8 × 9.81 = 10.20 N. The rule is right on A and B and wrong on C: the box is light (0.367 kg, needing only 4.69 N), but the rule assumes 0.8 kg. Grasp C is a safe grasp the rule rejects because the mass was not measured. The error comes from the missing mass, not from a bug in the arithmetic, and no choice of model fixes it.

Result 1: the honest model and the leaked model

For seed 1 the standardised weights of the five-column model are (offset, tilt, friction, force, slip) = −0.94, −0.18, 0.75, 0.63 and −5.35; without the slip column they are −1.57, −0.09, 1.49 and 1.54. The leaked column has a weight about six times the largest honest weight (0.94 in magnitude) in that model, a sign worth checking in any model. In a separate 5,000-grasp draw (seed 99) every successful grasp had a slip below 2.92 mm/s (mean 0.48) and every failed one above 3.00: the classes do not overlap at all, so one threshold separates them perfectly. A column that separates the labels perfectly, which no physical sensor measured before the event could do, is evidence of a leak.

Why does the leaked model fall to exactly the base rate at deployment? With slip = 0 the model sees the clean “held” signature for every grasp and predicts success each time, which is right 59.6% of the time. In a real system the slip column would simply not exist, and the model would have to be retrained without it. The honest accuracy was 0.823 all along.

Result 2: how much is the mass worth?

A rule that knows the true friction and mass reaches accuracy 1.000 on 5,000 grasps (seed 99), so the label is a deterministic function of the hidden values and the 17% of errors above are information, not noise. With a guessed mass the hand rule scores 0.767 (0.5 kg), 0.816 (0.65), 0.838 (0.8), 0.815 (0.95) and 0.780 (1.1 kg) on the same 5,000 grasps: the best guess is near the middle of the 0.3 to 1.3 kg range and the cost of a bad guess is up to 7 points. The learned model reaches the same accuracy without being told a mass because the logistic fit absorbs a typical one into its weights.

Result 3: friction noise, data size and a better feature

Three readings. Noisy friction measurement matters only a little at 10% (0.823 against 0.825 with no noise) and visibly at 30%. More data helps up to a few hundred grasps and then stops (0.823 at 300, 0.820 at 1,000), because the missing mass, not the sample size, is the limit. And feeding the model the physical combination μ̂ F cos(tilt) gives 0.827, a gain inside the intervals: the logistic model on the raw columns was already close to what these four numbers allow. All results are for this toy generator and its parameter choices; real objects vary in shape, pad compliance and surface, and the numbers would be different.

Engineering implications and limitations

Features that exist at decision time

Before using a column, ask when in the attempt it is measured. For each feature write down the sensor, the time it is read and whether the deployed robot reads it before it must decide. The usual leaks in grasping data are post-grasp readings (slip, lift height, tactile contact after the lift), the label’s own definition (the image after the lift, the weight reading from the scale after a pick), simulator state such as the true object pose, mass and friction, and the episode id that groups attempts at one object (lesson M2-T6). A simple test is the one used above: train with and without the column and ask what a threshold on the column alone achieves. A near-perfect single-feature separator is nearly always a leak.

Simulation-only features

If grasps are generated in simulation, the simulator knows the object’s exact pose, mass, friction and contact points. A model trained on those inputs cannot run on a robot that sees an image and a force reading. The remedies are to train on features the robot will have (images, depth, measured force), to add the noise the sensors really have (the 10% on friction here), or to use privileged information only in the critic or for labelling, never as policy input. The sim-to-real transfer problem is the subject of Module 10.

Data and labels

• Define success by an observable and long enough test: a lift, a shake, a hold for a few seconds. “The object left the table” is a weaker label than “it was still in the gripper after the shake”, and labels of different strictness cannot be pooled.
• Split by object, not by attempt (lesson M2-T6). Attempts at the same object share its mass and shape, so a random split inflates accuracy exactly like the leak above.
• Record the failure type (slip, collision, missed object, timeout) in addition to the binary label; a classifier cannot tell you which of them is the problem.
• Report the base rate with accuracy. With 58% successes, 0.596 for “always holds” is the number to beat. For a gripper, a missed failure and a missed success have different costs, so report both rates and not accuracy alone.
• Calibrate before acting on a probability: a predictor that says 0.9 should be right about nine times in ten. Selecting the grasp with the highest predicted success from many candidates makes the probability optimistic, the same selection effect as lesson M8-T3 for Q-values.

From a predictor to a grasp policy

A predictor rates a candidate grasp; something must still propose candidates and choose one. Systems such as Dex-Net sample many candidates and rank them with the learned score. Self-supervised systems such as Pinto and Gupta’s train on their own tries, and Levine and colleagues’ hand-eye coordination work runs a learned success predictor inside a closed loop that re-plans from camera images. Reinforcement-learning approaches (for example QT-Opt, Kalashnikov et al., 2018) learn the grasp directly from a binary success reward; that is a sparse-reward problem of the kind described in lesson M9-T2, and it has needed hundreds of thousands of real grasps. For a first system, a hand rule or a small classifier on honest features, wrapped by the safety layer of lesson M9-T4, is the baseline to beat.

What this toy cannot tell you

The toy grasp has five numbers and one cliff. Real grasps depend on the shape of the object, on how the fingers deform, on how the object moves during closing, and on dynamics after the lift. The simple lesson, that the model is limited by what its features contain and that a leak looks like success, does not depend on the toy; the specific accuracies do.

Common misconceptions

Engineering takeaways

1. Grasp success prediction is classification on features measured before the lift; the label is whether the object stayed in the gripper, and the base rate (here about 59%) is the baseline.
2. In this toy, four honest features give 0.823 held-out accuracy and a hand force-balance rule with a guessed 0.8 kg mass gives 0.833; the missing mass, not the model or the data size (0.823 at 300 grasps, 0.820 at 1,000), sets the limit.
3. A column measured after the lift (slip) lifts accuracy to 0.9997 and falls back to 0.596 when it is unavailable; a single feature that perfectly separates the labels is evidence of a leak.
4. Check every feature for when and by what sensor it is read, split by object rather than attempt, and report base rate, both error rates and calibration.
5. Analytic metrics, learned predictors and grasp policies have different data needs and failure modes; this module’s toy model is not a contact simulator, and the real systems it points to (Dex-Net, hand-eye coordination, QT-Opt) rely on far more data.

References and further reading

Primary sources only, cited from memory for what they are about, with no numbers taken from them. All numerical results in this lesson come from running the track’s toy generator.

• C. Ferrari and J. Canny, “Planning Optimal Grasps”, IEEE International Conference on Robotics and Automation (ICRA), 1992.
• J. Bohg, A. Morales, T. Asfour and D. Kragic, “Data-Driven Grasp Synthesis: A Survey”, IEEE Transactions on Robotics 30(2), 2014.
• L. Pinto and A. Gupta, “Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours”, IEEE International Conference on Robotics and Automation (ICRA), 2016.
• S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz and D. Quillen, “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, International Journal of Robotics Research 37(4-5), 2018.
• J. Mahler et al., “Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics”, Robotics: Science and Systems (RSS), 2017.
• D. Kalashnikov et al., “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”, Conference on Robot Learning (CoRL), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A grasp ends when the object is held. Lesson M9-T2 looks at what happens during contact: why forces change abruptly, why a stiff position controller can break things, how compliance and force feedback help, and why a sparse success reward makes the learning slow.

Related content

Nothing below is required to finish this lesson.

• Lab
  M9-L1: Predict Grasp Success

  Predict which grasps will hold from a few measured features, and find the feature that only works in simulation.