M9-T3 — Legged Locomotion with Reinforcement Learning

Learning objectives

After this lesson, you should be able to:

• Describe how a gait policy is trained in simulation: observations, actions, reward terms, termination on falls and the role of PPO and many parallel simulations.
• Compute the speed, energy, fall risk and reward of a gait in the toy gait-parameter model by hand, and state plainly what that model leaves out.
• Explain, with the measured results, how missing reward terms produce a shuffle, a hop or a fall, and what each added term changes.
• Say why a penalty weight can be too small even when it is nonzero, and why a gait labelled “walk” can still be unsafe.

The intuition

Suppose you want a biped to walk forward and write the obvious reward: be fast, do not waste energy, do not fall. You give the optimiser every possible gait and ask for the best one. With one weighting it finds a gait with foot clearance zero. The feet drag along the floor. That is not a bug in the optimiser: dragging costs no energy in your formula, and nothing in the reward mentions the floor. Change the weights and it finds a hop on two feet, or sprints at a speed where the walker falls on every trial and reports a very high score on the way down.

The figure shows four weightings and the gait each one selects. Only the last, with a term for every property you actually care about, selects a gait you would call walking. Speed 2.97 sounds impressive until you read the fall probability beside it: 1.00. The reward counted the speed and underweighted the fall.

Real locomotion training has the same structure and a much larger space. The designers of the reward terms look at the learned gait, find the loophole it exploits, and add a term. The purpose of this lesson is to make that cycle visible, with numbers, in a model small enough to check by hand.

The mathematics

How a gait policy is trained in simulation (the general recipe)

The pipeline, as I understand it from the literature (Tan et al., 2018; Hwangbo et al., 2019; Lee et al., 2020; Rudin et al., 2021), is:

1. Simulate the robot and terrain in a physics engine, often thousands of copies in parallel on a GPU, with randomised parameters (friction, mass, motor strength, push disturbances) so the policy does not fit one simulator (Module 10).
2. Observe what the robot can sense on hardware: body orientation and angular velocity from the IMU, joint angles and velocities from encoders, the commanded velocity, and often the previous action. Not the true contact forces or terrain height unless a sensor provides them.
3. Act with joint position targets (or small offsets from a nominal stance) that a low-level PD controller tracks, at a rate of tens of Hz. This is the same split between a learned policy and a fast classical controller described in lesson M9-T2.
4. Reward a weighted sum of terms: tracking the commanded velocity, minus penalties for torque or energy, joint limits, body tilt, foot slip and action jitter, sometimes a foot-clearance or contact-timing term. The weights are the designer’s main knob (lesson M9-T6).
5. Terminate the episode when the body hits the ground, and train with PPO (lesson M6-T5). Termination makes a fall end the stream of reward, which acts as a penalty if rewards are positive and as a relief if they are negative: another place where a designer’s choice is a reward term (lesson M5-T5).
6. Deploy the fixed network on the robot, ideally after a staged test (lesson M10-T5); the gap between simulator and hardware is where most real surprises come from (Module 10).

I describe this at the level of what such systems share. I do not claim to know the observation list, reward weights or training time of any particular robot beyond the point that the papers above report policies trained in simulation and transferred to hardware.

A toy: a gait is four numbers

The track’s RLCore.gaitMetrics describes a two-legged walker by a gait g = (f, L, phase, h): step rate f in steps per second (0.6 to 3.6), step length L in metres (0.05 to 0.7), the phase between the feet (0: both land together, a hop; 0.35: a limp; 0.5: alternating steps), and the foot clearance h in metres (0 to 0.12). From these it computes five numbers:

where σ is the logistic function. The reward is a weighted sum with weights w:

These formulas are my own hand-built guess at the shape of the trade-offs (energy grows with the square of speed, a faster walker is closer to losing balance, alternating gaits are more stable than hops). They are not fitted to a robot or derived from mechanics. They contain no joints, no contact forces, no terrain and no controller, so nothing about their numbers carries over to hardware. What is real is the logic of the experiment: a fixed reward, an exhaustive optimiser and the gait it chooses. RLCore.bestGait(w) evaluates the reward on a grid of 1,680 gaits and returns the best; RLCore.gaitKind labels it: fall (fall probability above 0.2), hop (phase below 0.25), shuffle (clearance below 0.04, feet dragging), limp (phase below 0.45), creep (speed below 0.4) or walk.

f, L

Step rate (steps/s) and step length (m); speed = f L.

phase

0 for a hop, 0.35 for a limp, 0.5 for alternating feet.

h

Foot clearance in metres; below 0.04 m the feet scuff.

fall

Toy fall probability, a logistic function of speed relative to a critical speed that depends on the phase.

energy

Toy cost of transport; clearance costs energy, which is why h = 0 is “free”.

w

The five reward weights (speed, energy, fall, scuff, asym).

A worked example

All gait numbers are outputs of RLCore.gaitMetrics, gaitReward and bestGait in Node. The search is exhaustive over a fixed grid, so every result is exact and repeatable, with no seeds for the four named weightings. Two sweeps below draw random weights (seeded) and are labelled.

By hand: one gait, start to finish

Take the walk found for the fourth weighting: f = 1.8, L = 0.7, phase 0.5, h = 0.06. Speed = 1.8 × 0.7 = 1.26. For phase 0.5, sin²(π/2) = 1, so v_crit = 0.7 + 1.5 = 2.2 and the fall probability is σ(10 × (1.26 / 2.2 − 1.1)) = σ(−5.27) = 0.0051. Energy = 0.4 + 2.5 × 1.26² + 0.3 × 1.8 + 25 × 0.06 × 1.8 = 0.4 + 3.969 + 0.54 + 2.7 = 7.609. Scuff: h = 0.06 ≥ 0.04, so 0. Asymmetry 0. With w = (speed 2, energy 1, fall 2, scuff 2, asym 1) the reward is 2 × 1.26 − 7.609 / 5 − 2 × 0.0051 = 2.52 − 1.522 − 0.010 = 0.988, which matches the program’s 0.99.

Now the same gait with the feet dragging (h = 0): speed and fall are unchanged, the clearance term of the energy vanishes (25 × 0 × 1.8), so energy = 4.909, and scuff = 1. Under a reward with no scuff term, w = (speed 1, energy 1, fall 0.5, scuff 0, asym 0), the dragging gait scores 1.26 − 4.909 / 5 − 0.5 × 0.0051 = 0.276 against 1.26 − 7.609 / 5 − 0.0026 = −0.264 for the lifted foot: dragging wins by 0.54, exactly the energy saved (2.7 / 5). The optimiser’s favourite under those weights is a slightly slower shuffle, f = 1.4, L = 0.7, h = 0, speed 0.98, reward 0.335. Nothing is broken: the formula rewarded dragging.

Four weightings, four gaits

Reading across: the first weighting values speed highly and its fall penalty (0.5 × 1.00) is small next to the speed reward (3 × 2.97 = 8.9), so it takes the fall; its reward of 6.42 is the highest of the four, which shows that a high reward is not a good gait. The second weighting cannot afford much speed and has no asymmetry penalty, so it picks the cheapest gait, a hop at a very low step rate. The third has no scuff term and shuffles. Only the fourth gets a walk. Cross-evaluating the four chosen gaits under the fourth reward gives walk 0.99, hop −0.31, shuffle −0.69 and fall −3.06: the walk is best under the reward that closes every loophole, and every “hack” is worse under it. Under the third weighting, though, the walk scores −0.264 against 0.335 for the shuffle.

Which term closes which loophole

Three lessons from the sweeps. Fall penalties of 0, 0.5 and 1 leave the fall in place when the speed weight is 3; a penalty of 2 moves the answer to a walk, but that walk still has a 19% fall probability, just under the 0.2 line the label uses, which is why a gait called “walk” may not be safe: the label is a threshold on a number. Closing one hole can open another: an asymmetry weight fixes the hop but the walk it picks is slow, and an energy weight of 6 turns the walker into a creeping, slightly scuffing shuffle (the foot clearance costs energy). And the answer changes in steps: between a scuff weight of 0 and 0.25 the optimum jumps from shuffle to hop with nothing in between.

How often is the answer a walk?

Seeded with SLAMCore.rng(7): 500 random weightings with each of the five weights uniform in [0, 3] gave walk 233, shuffle 126, creep 71, fall 49, hop 21. A walk was found for 46.6% of weightings, so more than half of arbitrary reward settings give a gait nobody wanted. With the scuff and asymmetry weights set to 0 and the other three weights random in [0, 3], a further 500 draws gave 444 shuffles and 56 falls, and no walk at all. In this toy, if the reward does not mention the properties of a good gait, the optimiser does not find one.

What the toy simplifies. The gait space is four numbers; the optimiser is an exhaustive search and not PPO; the fall probability and energy are guesses; there is no terrain, no balance, no sensing, no robot. The real process is noisier and the loopholes are more varied (for example, a learned policy may vibrate its legs, lean on a wall, or hold a pose to avoid a penalty). The numbers above are exact for this toy and say nothing about the speed, energy or stability of a real robot.

Engineering implications and limitations

A workflow for locomotion rewards

1. Write down the behaviour you want in sentences (“walk forward at the commanded speed, with feet lifted, upright, without jerks”) and give each clause a term. A property with no term is a property you did not ask for.
2. Train, then watch the gait and read the per-term reward curves (lesson M6-T7). A rising total reward with a falling or flat term is where to look first.
3. Name the loophole (the shuffle, the hop, the sprint-and-fall) and add a term for it or a constraint, not a patch on the weights alone. Check the other loopholes did not open (the asymmetry sweep above).
4. Check that the weights are in the right ratio. In the toy, a fall penalty of 1 against a speed reward of 3 does not stop a fall; 2 does but leaves a 19% fall probability. Evaluate fall frequency directly (lesson M1-T5), not through the reward.
5. Test on hardware only after the policy passes a robustness test in simulation with changed friction, mass and pushes (Module 10), under the safety layer of lesson M9-T4.

Observations and actions in practice

• Include the previous action and a smoothness penalty so the output does not jitter at the control rate. Jitter that is harmless in simulation can damage motors.
• Use observations the robot has on hardware, with the sensor noise and delay it has (lesson M9-T1 on features that exist only in simulation). A policy that sees the true terrain height in simulation cannot run on a robot that does not.
• Output joint position targets for a PD controller around the standing pose, not raw torques, so that a random early policy does not fold the robot; it also limits what the policy can command.
• Terminate on a fall and on joint-limit or self-collision events, and decide deliberately how the termination interacts with the sign of the reward (lesson M5-T5).
• Curriculum on terrain and command range (lesson M9-T5) reduces the chance that the first thing the policy finds is a degenerate gait.

Evaluating a gait

Report fall rate per metre or per minute with an interval over seeds and terrains (lesson M1-T5), energy per distance, the commanded-versus-achieved speed, and a video of the worst episodes. Do not report the reward alone: in the toy, the highest reward (6.42) belonged to the gait that fell on every trial. For a deployed robot, the safety layer and rollout of lessons M9-T4 and M10-T5 apply.

Common misconceptions

Engineering takeaways

1. A gait policy is trained in simulation with observations the robot has on hardware, joint-target actions for a low-level controller, a reward that sums velocity tracking and penalties, termination on falls, and PPO with many parallel simulations.
2. The toy gait-parameter model (f, L, phase, h) is not a physics simulator and makes no claim about real robots; it is exact enough to show what an optimiser does with a given reward over 1,680 gaits.
3. Without a scuff term the best gait is a shuffle (h = 0, speed 0.98), without an asymmetry term it can be a hop (speed 0.53), and with a speed weight of 3 and a fall penalty of 1 or less it is a fall (speed 2.97, fall probability 1.00, the highest reward at 6.42).
4. With the scuff and asymmetry weights at zero, 500 random weightings gave 444 shuffles, 56 falls and no walk; with every term free, 233 of 500 gave a walk.
5. Fix a reward by adding a term for each property you want, check fall frequency directly and watch the gait; a label such as walk can hide a 19% fall probability. Lessons M9-T6 and lab M9-L2 pick up the design of locomotion observations and rewards.

References and further reading

Primary papers for learning legged locomotion in simulation and transferring it to hardware. I cite them for what they showed in general terms; the numbers and the model in this lesson are the track’s own.

• J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.
• J. Tan et al., “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots”, Robotics: Science and Systems (RSS), 2018.
• J. Hwangbo et al., “Learning Agile and Dynamic Motor Skills for Legged Robots”, Science Robotics 4(26), 2019.
• J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun and M. Hutter, “Learning Quadrupedal Locomotion over Challenging Terrain”, Science Robotics 5(47), 2020.
• N. Rudin, D. Hoeller, P. Reist and M. Hutter, “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, Conference on Robot Learning (CoRL), 2021.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A gait policy that looks right in simulation is not yet a safe robot. Lesson M9-T4 puts a learned policy inside a classical controller with limits, a monitor and a fallback, and says what each part guarantees and what it cannot.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T2: Contact-Rich Manipulation

  Explain why contact makes learning and control hard, and how compliance and force feedback help.

• Lab
  M9-L2: Shape a Gait Reward

  A simulated walker shuffles, hops or falls depending on the reward weights: choose weights that walk.