M9-T2 — Contact-Rich Manipulation

Learning objectives

After this lesson, you should be able to:

• Explain why contact makes dynamics discontinuous or very stiff, and why small position errors become large forces.
• Describe impedance (compliance) control and admittance (force feedback) control in words, and compute the steady contact force of a series spring by hand.
• Explain why a sparse success reward makes contact-rich tasks hard to learn, with measured effects of a tighter goal on PPO.
• Say what a learned policy can take from force and torque sensing and a classical compliant controller, and what remains unchecked.

The intuition

Slide a fingertip towards a wall. In free space the force on it is zero, whatever the position. The moment it touches, the wall pushes back with a force that grows with the depth: for a stiff wall (20,000 N/m is 20 N for a millimetre), the force goes from nothing to a large value inside a hair’s breadth of position. A controller that only knows positions has to know where the wall is to the millimetre to produce the force it intends.

The figure shows three ways of commanding the same push. A stiff position controller (gain equal to the wall’s) is told to go 3 mm past the wall: the steady force is 30 N. If the wall is really 2 mm further away than modelled, the same command now gives 10 N. A compliant controller with a gain of 500 N/m gives 1.46 N and 0.49 N: smaller and still sensitive, but the error is in newtons, not tens of newtons. A controller that measures the force and moves until it reads the target gets 5.0 N in both cases, whatever the wall position. Compliance trades stiffness for robustness to position error, and force feedback goes further by controlling the force itself.

For learning, the same cliff is a problem for a different reason. The reward, the observation and the dynamics all change abruptly at contact, and a policy that has not yet touched the object receives no information about touching it. Both effects are measured below.

The mathematics

Contact as switching dynamics

Take a fingertip at position x moving towards a wall at x_w. The simplest model of a rigid wall treated as a stiff spring has a force that depends on the sign of the penetration:

The force is continuous at the wall but its slope jumps from 0 to k_w. A real impact adds energy loss, friction (sticking or sliding, which is again a switch) and a change of the set of contacts. Planning, model learning and policy search all have to cope with a function that is smooth in pieces and changes between them. For a rigid contact the acceleration is not even defined at the instant of impact, which is why simulators use springs, impulses or optimisation to approximate it. The same cliff is present in Scout’s world: the reward of one step is −0.05 until the robot is within 0.95 m of the pillar centre, and −5.05 as soon as it is inside, a jump of 5 in one step (RLCore.step).

A rule of thumb that follows from the spring model: a mass m approaching a wall of stiffness k_w at speed v, with no damping, compresses the spring until all its kinetic energy is stored, so the peak force is v √(k_w m). The force grows linearly with approach speed, and a stiffer wall is harder.

Impedance control: choose how the robot gives way

An impedance controller makes the robot behave like a spring and a damper around a desired position x_d:

It does not insist on the position. With a small K the robot yields when the world pushes back, and the force it applies against a surface is K times the position error (more exactly, a series of two springs, below). Hogan (1985) developed the idea that the robot should regulate the relation between motion and force at its contact (its dynamic “impedance”) and not the position or the force alone. The gains K and D are the design variables. They are also natural parameters for a learned policy to output, which is the idea behind several of the recent contact-rich learning methods: the policy chooses a target and a stiffness and a low-level compliant controller does the force handling at a high rate.

Admittance control and force feedback

An admittance controller does the opposite mapping: it reads the force and outputs a motion. In its simplest form the position moves at a speed proportional to the force error:

With a target of 5 N the fingertip advances until the sensor reads 5 N and then stops, whatever the wall position. The price is the sensor, the delay in the loop, and a stability limit: a stiff environment turns the force error into a large position step. In a discrete loop with step dt the force error is multiplied by k_w dt / B each cycle, so the loop settles if that factor is below 1, overshoots while it is below 2, and oscillates (bounces on and off the wall) at 2 and above. Raibert and Craig (1981) proposed controlling force along the directions in which the surface constrains motion and position along the others (hybrid position and force control); Mason (1981) analysed the geometry behind this split.

k_w

Stiffness of the environment (wall spring), N/m.

K, D

Stiffness and damping of the impedance controller.

x_d, x_w

Commanded position and the wall’s actual position.

B

Admittance gain: damping that turns a force error into a speed, N s/m.

F_target

Force the admittance controller regulates to.

goalR

Radius of the success region in the Scout stand-in for a tight tolerance, in metres.

Why a sparse reward is hard here

For insertion, wiping or assembly, the natural reward is “1 when the task is done”, and that region is small compared with the space the robot explores. A random exploration policy almost never finds it, and the learner gets no gradient. Lesson M5-T3 describes sparse and dense rewards in general; contact adds that the informative states (touching, aligned, inserted) are a vanishing part of the state space. The remedies are the ones of Module 5 (shaping, lesson M5-T2) and the ones of Module 3: start from demonstrations (lesson M3-T2), use a compliant controller as the base policy and learn a residual correction on top of it, as in the residual reinforcement learning of Johannink et al. (2019), or use a curriculum (lesson M9-T5).

A worked example

By hand: two springs in series

The fingertip is driven by an impedance controller with stiffness K and meets a wall of stiffness k_w = 20,000 N/m. At rest the controller spring and the wall spring carry the same force, and their deflections add up to the total error, the distance from the commanded position to the wall: Δ = x_d − x_w. So F = K (Δ − p) and F = k_w p, where p is the penetration, which gives

With Δ = 3 mm: for K = 500 N/m, K k_w / (K + k_w) = 500 × 20,000 / 20,500 = 487.8 N/m, so F = 487.8 × 0.003 = 1.463 N and the penetration is 1.463 / 20,000 = 0.073 mm. For K = 2,000, the series stiffness is 1,818.2 N/m and F = 5.455 N. For K = 20,000 it is 10,000 N/m and F = 30.0 N, a penetration of 1.5 mm. If the wall is really 2 mm further away than assumed, the error is 1 mm, and the three forces become 0.488 N, 1.818 N and 10.0 N. The stiff controller changes by 20 N for a 2 mm error in where the wall is; the compliant one by 1 N.

I checked all six by simulating a 1 kg fingertip with the controller gains of 500 (damping 60), 2,000 (damping 120) and 20,000 (damping 300) against the wall for 3 s with a 0.5 ms step: the steady forces were 1.463, 5.455 and 30.000 N, and 0.488 and 10.000 N for the shifted wall and K = 500 and 20,000, matching the formula to three decimals.

By hand: the impact force and the admittance limit

A 1 kg fingertip hits the same wall at 25 mm/s: the peak force is v √(k_w m) = 0.025 × √20,000 = 0.025 × 141.4 = 3.54 N. At 100 mm/s it is 14.14 N. A simulation with a 10 microsecond step gave 3.536 and 14.142 N. Speed matters linearly, which is why contact-rich policies slow down near contact.

The admittance controller with F_target = 5 N and B = 200 N s/m, 1 ms step, started 5 mm in front of the wall. The steady penetration is F_target / k_w = 5 / 20,000 = 0.25 mm, and the simulated force settled at 5.000 N for a wall at 0.100 m and for one at 0.102 m. The loop factor k_w dt / B is 20,000 × 0.001 / 200 = 0.10. With B = 15 (factor 1.33) the force overshot to 6.7 N and then settled at 5.00 N; with B = 10 (factor 2.00) it alternated between 0 and 10 N forever (the fingertip bouncing on and off the wall); with B = 5 (factor 4.00) it reached 20 N and kept bouncing. Force feedback is only as good as the stability of its loop, in a toy model of a perfectly rigid, perfectly measured wall with no sensor delay; a real sensor, a real delay and a real compliant wall move the numbers.

Measured: a tighter goal slows learning

Scout’s hall is not a contact task, but it has what matters here: a small region that pays +10. I shrank the goal radius from the default 0.5 m (a disc of area 0.785 m², 1.31% of the 60 m² hall) to 0.25 m (0.33%) and 0.1 m (0.052%) and trained the track’s PPO (lesson M6-T5; defaults: 60 iterations of 12 episodes, 32 hidden units, five seeds). The table reports the share of training episodes that reached the goal over the last 10 iterations of each seed (these are the stochastic training policy’s episodes, not a held-out evaluation).

Random actions never reach the goal in 2,000 episodes at any of the three radii, so exploration by chance contributes nothing here. The learner nevertheless improves in most seeds, because the −0.05 per step and the crash penalty give a learning signal, and the size of the goal changes how often it gets lucky. Mean training success falls from 0.685 to 0.423 to 0.190 as the radius shrinks, with five seeds and a large seed-to-seed spread (seed 3 never learned at any radius; five seeds is too few for intervals I would trust). I read it as a direction and not a rate: the tighter the success region, the more seeds fail. The experiment does not say how an insertion task with real contact would behave.

Engineering implications and limitations

What to sense

Position and vision tell the robot where things are; they do not tell it that it is pushing too hard. A wrist force and torque sensor, motor current or tactile pads provide the direct signal. The cheapest lesson from this chapter is: feed force into the controller, not only into the learner. Learned policies built on force and touch data exist (for example Lee et al., 2019, which combines vision and touch for insertion-type tasks), and I describe them only at that level.

A division of labour that works

1. A high-rate compliant controller (impedance or admittance, hundreds to thousands of Hz) holds the contact and limits force. It is classical, analysable and slow to change.
2. A learned policy at a lower rate (tens of Hz) sets the target pose, the stiffness or the force target, or adds a residual correction, from observations that include force.
3. A safety layer (lesson M9-T4) limits force, speed and workspace whatever the policy asks, and stops on a force spike.
4. Logging includes force and the controller’s gains, so that a failed insertion can be diagnosed from the force trace and not guessed from video.

Practical points

• Reduce speed near contact: the peak impact force scales with approach speed (3.54 N at 25 mm/s, 14.14 N at 100 mm/s in the example).
• Randomise the position of the surface and the stiffness in simulation (Module 10), because the example shows how much the force depends on a millimetre of error.
• Reward the contact event with a staged reward: reaching, touching, aligning, inserting, with an effort or force penalty, rather than one reward at the end (lessons M5-T2 and M5-T4 on shaping and reward hacking).
• When a policy that works in simulation fails on hardware at the moment of contact, check the contact model first: stiffness, friction and sensor delay are the usual suspects.
• Start learning from a safe base: demonstrations (lesson M3-T2), or a compliant controller plus a small residual.

What this lesson did not test

No friction, no rotation, no object dynamics, no sensor noise or delay, one stiffness, and a toy search for the effect of sparse reward on a navigation task. The two methods are described at the level of a textbook idea; real implementations deal with six-dimensional forces, the robot’s own dynamics, and gain scheduling that I did not model.

Common misconceptions

Engineering takeaways

1. Contact switches the dynamics: zero force in free space, a steep spring on touching; reward, observation and force all change abruptly, which is hard for planning, model learning and policy search.
2. With a wall of 20,000 N/m, commanding 3 mm past it gives 30.0 N with a stiff position controller and 1.46 N with an impedance controller of 500 N/m; a 2 mm error in the wall position moves these to 10.0 N and 0.49 N, and a force-target controller held 5.0 N both times.
3. Impedance control makes the robot behave like a spring and damper and trades stiffness for robustness; admittance control turns a measured force error into motion, and its loop is only stable if the gain times the stiffness is small enough (factor below 1 settled, 2 or more bounced).
4. A sparse success reward is hard when the success region is small: random exploration reached it 0 times in 2,000, and PPO training success fell from 0.685 to 0.423 to 0.190 as the goal radius shrank from 0.5 to 0.25 to 0.1 m (five seeds, a toy stand-in).
5. A practical architecture combines a high-rate compliant controller, a lower-rate learned policy and a safety layer (lesson M9-T4), and records force so failures can be diagnosed.

References and further reading

Primary sources on impedance, hybrid position and force control, compliance, and learned residual and multimodal contact policies, cited for the ideas they introduced. All numbers in this lesson come from the track’s toy models.

• N. Hogan, “Impedance Control: An Approach to Manipulation: Part I - Theory”, Journal of Dynamic Systems, Measurement, and Control 107(1), 1985.
• M. H. Raibert and J. J. Craig, “Hybrid Position/Force Control of Manipulators”, Journal of Dynamic Systems, Measurement, and Control 103(2), 1981.
• M. T. Mason, “Compliance and Force Control for Computer Controlled Manipulators”, IEEE Transactions on Systems, Man, and Cybernetics 11(6), 1981.
• T. Johannink et al., “Residual Reinforcement Learning for Robot Control”, IEEE International Conference on Robotics and Automation (ICRA), 2019.
• M. A. Lee et al., “Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks”, IEEE International Conference on Robotics and Automation (ICRA), 2019.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

So far the robot has had a hand or a finger. Lesson M9-T3 gives it legs: a gait policy trained in simulation, its observations, actions and reward terms, and how an optimiser finds the loopholes in a reward (a shuffle, a hop, a fall) that nobody intended.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T1: Learning to Grasp

  Describe grasp success prediction and grasp policies, and what the data and the sensor must provide.

• Lab
  M9-L1: Predict Grasp Success

  Predict which grasps will hold from a few measured features, and find the feature that only works in simulation.