M9-T6 — Observations and Rewards for Locomotion

Learning objectives

After this lesson, you should be able to:

• State what each of five reward terms (speed, energy, fall, foot clearance, symmetry) is for, and give a weight set that exploits the absence of each in the toy gait model.
• Compute the metrics and the reward of a gait by hand from the model’s formulas, and explain why only the ratios of the weights matter.
• Separate the observations a gait policy can have on the robot (proprioception, previous actions, a command, history) from privileged simulation information, and say how a policy is trained with the latter.
• Report measured effects of observation noise, delay and smoothing on a policy, and say what a history buys and costs.

The intuition

Suppose you pay a walker only for forward speed and a little for saving energy. The optimiser looks for the gait that earns the most, and the simulator’s physics decide what that is. If dragging the feet is cheap, it drags them. If bounding on both feet is cheap, it hops. If falling forward at speed costs nothing, it falls forward, fast, because for an episode of fixed length a fall is a perfectly good way to cover distance. Each behaviour is a correct solution of the problem you wrote down. The practitioner’s response is to add a term that makes that particular cheat expensive, and the reward grows into a list of patches.

The figure shows this in the toy model. Four weight sets, and for each the best gait (found exactly by searching the whole grid of 1,680 gaits). The walk row has all five terms; the shuffle row has no clearance term, so scuffing the floor is free; the hop row has no symmetry term, so both feet landing together is allowed; the fall row has a big speed weight and no scuffing or asymmetry term, and the optimum is a gait that is certain to fall. The simple reading is that the reward is not a description of "walking"; it is a set of prices, and the optimiser buys whatever is cheapest per unit of the thing it is paid for.

Observations are the other half. The policy can only react to what it senses: if the weights reward foot clearance and the policy has no sense of its feet or the ground, it cannot do it reliably. If it is trained with information that the real robot will not have, it will work in simulation and fail on hardware, which is the lesson-M9-T1 leaked-feature problem again, at larger scale.

The mathematics

The toy gait model

A gait is g = (f, L, φ, h): step frequency f in steps per second (0.6 to 3.6), step length L in metres (0.05 to 0.7), phase φ between the feet (0 both feet together, i.e. a hop; 0.5 alternating; 0.35 a limp), and foot clearance h in metres (0 to 0.12). The model’s metrics are:

(h_eff is the clearance, raised to at least 0.06 for a hop.) The reward is a weighted sum with five weights w = (w_speed, w_energy, w_fall, w_scuff, w_asym):

and RLCore.bestGait(w) returns the gait that maximises R on a grid. RLCore.gaitKind labels a gait fall (fall probability above 0.2), hop (φ below 0.25), shuffle (h below 0.04, feet dragging), limp, creep (speed below 0.4) or walk. Multiplying all weights by a constant multiplies the reward and leaves the best gait unchanged (checked: weights ×3 give the same walk), so only the ratios of the weights matter in this model; in real training the scale also matters through the learning rate and the value-function normalisation.

What a real reward looks like

I describe the kind of terms used in published legged-locomotion training (for instance Hwangbo et al. and Rudin et al.) in general words and not as exact formulas, because the weights and forms vary between papers: a term that rewards tracking a commanded forward and turning velocity, terms that penalise joint torques or power, a term that penalises fast changes between consecutive actions (smoothness), penalties on body height and orientation error, a term on foot clearance or contact timing, and termination (a negative reward and the end of the episode) when the body falls or touches the ground. The toy model has a compressed version of five of them.

speed

Forward speed of the gait, f × L (times 1.25 for a hop).

fall

Probability of falling, a sigmoid in the ratio of speed to a phase-dependent critical speed.

scuff

Fraction of the clearance shortfall below 0.04 m: 1 when the feet drag, 0 above 0.04 m.

asym

Departure of the phase from alternating steps (0 for φ = 0.5, 1 for a hop).

observation o_t

What the policy sees at step t, for a real robot: joint angles and velocities, body orientation and angular velocity from an IMU, the last action, a command, optionally a short history of these.

privileged information

Quantities available in simulation and not on the robot: terrain height, friction, body linear velocity in the world, contact forces, payload mass.

A worked example

Every number comes from RLCore.gaitMetrics, gaitReward and bestGait run in Node, and, for the observation experiment, from 5 seeds of 30 Scout starts. The model is a toy: its numbers say what the model does, and not what a robot does.

By hand: why the walk beats the shuffle (and when it does not)

The walk gait is f = 1.8, L = 0.7, φ = 0.5, h = 0.06. Speed = 1.8 × 0.7 = 1.26. v_crit = 0.7 + 1.5 × sin²(π/2) = 2.2, so fall = sigmoid(10 × (1.26/2.2 − 1.1)) = sigmoid(−5.27) = 0.005. Energy = 0.4 + 2.5 × 1.26² + 0.3 × 1.8 + 25 × 0.06 × 1.8 = 0.4 + 3.969 + 0.54 + 2.7 = 7.609. Scuff = 0 (h ≥ 0.04) and asymmetry = 0. Under w = (2, 1, 2, 2, 1): R = 2 × 1.26 − 7.609/5 − 2 × 0.005 − 0 − 0 = 0.988.

The shuffle gait is f = 1.4, L = 0.7, φ = 0.5, h = 0: speed 0.98, fall 0.001, energy 0.4 + 2.5 × 0.98² + 0.3 × 1.4 = 3.221 (no clearance cost), scuff 1. Under the same weights R = 1.96 − 0.644 − 0.003 − 2 = −0.687, much worse than the walk’s 0.988. Under w = (1, 1, 0.5, 0, 0) (no scuff term) R_shuffle = 0.98 − 0.644 − 0.001 = 0.335 and R_walk = 1.26 − 1.522 − 0.003 = −0.264. The shuffle now wins by 0.6: foot clearance costs 25 × 0.06 × 1.8 = 2.7 energy units for the walk, and nothing in the reward pays for it. The optimiser is not malfunctioning; it is minimising a cost that you did not know was a cost.

Four exploits

The hop exists in the model because a hop gets a 1.25 speed bonus and a 0.7 energy factor, at the price of a much lower critical speed (0.7 against 2.2 for alternating steps); that is a design choice of the toy, which gives the symmetry term something to do. With the hop weights plus an asymmetry weight of just 0.1, the best gait becomes a (slow) walk. It is not that hopping is bad; it is that nothing said otherwise.

What each term does: remove one, or turn it up

Start from the walk weights (2, 1, 2, 2, 1) and set one weight to zero:

Sweeping a single weight from the walk weights shows how sharp the transitions are. Scuff: a weight below 0.45 gives h = 0 (full shuffle), between 0.45 and 1.3 gives h = 0.03 (quarter scuff, still a shuffle in the model’s classification), and from 1.35 up gives the full clearance h = 0.06. Speed: from 0.5 the best gait is a walk, with speed 0.42 at weight 0.5 to 1.0, 0.70 at 1.1, 0.98 at 1.4, 1.26 at 1.7 and 1.82 at 2.6 (so the speed asked for rises smoothly); from weight 3.8 it flips to a fall at speed 2.97. Energy: at weight 0 a fall; at 0.15 a walk with fall risk 0.19; at 2 to 4 a slow walk at 0.42; at 8 a shuffle at 0.24 (too slow to be worth lifting the feet). A fall weight of 0 gives a fall, but with the energy term present anything from 0.5 to 4 gives the same kind of walk, because the energy penalty already keeps speed below the cliff: the fall term is close to redundant here. With speed weight 3 and no energy or scuff pressure it is not: a fall weight of 0.5 or 2 still gives a fall, 4 gives a shuffle at speed 1.95 (fall risk 0.11), 8 a shuffle at 1.82.

These results are exact for the toy model and for the grid of 1,680 gaits; a real policy with a neural network and a physics simulator finds different cheats, and the same sentence applies to them: the optimum depends on every price in the list. The fall penalty is also the clearest case for the choice between a reward term and a constraint (lesson M5-T5): a probability of falling is a safety quantity, and "pay 0.5 per unit of risk" is a trade you probably did not mean to offer.

Observations: noise, delay and smoothing on Scout

The toy gait model has no observations, so I use Scout’s reach task to measure how a good policy (the expert) degrades with a corrupted observation. The corruption is an independent Gaussian error of standard deviation σ per step on the estimated robot position, which shifts the goal vector and the pillar vector together. Success out of 150 (5 seeds × 30 starts):

Averaging the last three observations is the simplest way to use a history, and here it made things worse at the largest noise (11 crashes against 5): the average of three measurements taken while moving at 1 m/s lags the true position by about 0.25 m, which costs more than the noise it removes. A delay of one step (0.25 s) was the worst. A history is useful when it is used to estimate what is hidden (a velocity from consecutive positions, a contact state, a payload), not when it is just averaged. What I did not test: learned history encoders, or noise correlated in time, which is the realistic kind.

Engineering implications and limitations

Choosing the observation

The policy should get what is cheap to measure on the robot and informative about the next action. For a legged robot the usual deployable set is proprioception (joint angles and velocities from the encoders, body orientation and angular velocity from the IMU), the previous action, and the command (the target velocity or heading); often a short history of these, so that velocities, contacts and some of the unobserved dynamics can be inferred. Whether to add exteroception (a height map, a camera) depends on the task: proprioception-only policies have been reported to handle rough terrain (Lee et al.), while gaps and similar hazards usually need vision.

Privileged information in simulation

Two common ways to profit from what only the simulator knows without deploying it. Asymmetric actor-critic: the critic, used only for training, sees the full simulator state while the actor sees only deployable observations (Pinto et al.). Teacher-student: a teacher policy with privileged inputs is trained first; a student policy with only deployable inputs and a history is trained to reproduce the teacher’s actions, so the history learns to infer the hidden quantities (Lee et al. for terrain; Kumar et al., RMA, for an adaptation module that infers an encoding of the environment parameters from recent history). The safe rule is the same as in lesson M9-T1: any input to the actor must exist on the robot at the time of the decision. Check each input by asking "could I measure this online, with this noise and this delay?". Everything else belongs to the critic or the teacher.

Designing the reward

• Write down what each term is for and the behaviour that term does not forbid. If you cannot name the cheat it prevents, drop it or keep it small.
• Prefer tracking to maximising. "Match the commanded speed" pays nothing for more; "maximise speed" pays for every unit, up to a fall (weights above 3.8 at the walk settings).
• Treat safety as a constraint where you can: a termination on falling, or a constrained optimisation, rather than a price (lesson M5-T5).
• Check the optimum, not the weights: run the trained policy, look at the gait, the foot contacts and the energy use; log each term per episode so that a term that dominates, or one that is always zero, is visible (lesson M6-T7).
• Change one term at a time, over several seeds, and report the behaviour with the metrics that were not in the reward. Each term you add changes every other term’s effective price.
• Expect the simulator to be part of the reward. Contact models, actuator limits and friction determine which gaits are cheap; the cheats that remain are the ones the simulator allows (lessons M10-T1 and M10-T2).

Honest limits

The toy model has five terms, four parameters and a grid, and its formulas were chosen to produce these behaviours; it does not show how reinforcement learning finds, or fails to find, such gaits, how long it takes, or how contact physics and actuators change the picture. The observation experiment is on Scout, a point robot, with white noise. Use these results as a vocabulary for what to check on a real gait, and not as a prediction.

Common misconceptions

Engineering takeaways

1. A gait reward is a list of prices; the optimiser buys the cheapest behaviour that earns reward, so every term is a patch for a cheat someone has seen (shuffle, hop, fall).
2. In the toy gait-parameter model (not a physics simulator), the weights (1, 1, 0.5, 0, 0) give a shuffle, (0.5, 0.5, 0.5, 0.5, 0) a hop, (3, 0.5, 0.5, 0, 0) a fall at speed 2.97, and (2, 1, 2, 2, 1) a walk at speed 1.26 with fall risk 0.01; only the ratios matter.
3. Terms interact: removing the scuff term gives a shuffle, removing the energy term a fall risk of 0.28, removing the asymmetry term changes nothing at the walk weights but removes the hop at the hop weights.
4. The policy should observe what the robot can measure at decision time (proprioception, last action, command, history); simulator-only information belongs to the critic or a teacher (asymmetric actor-critic, teacher-student), never to the deployed actor.
5. A history helps when it estimates something hidden and hurts when it only adds lag: on Scout, averaging three observations gave 11 crashes against 5 for the raw reading at σ = 0.4 m, and a one-step delay 25.

References and further reading

The locomotion papers are the primary sources for the reward and observation practices I describe only in general words; I do not reproduce their reward formulas or weights. The gait model and all numbers in the example are my own toy.

• J. Hwangbo et al., “Learning agile and dynamic motor skills for legged robots”, Science Robotics 4(26), 2019.
• J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun and M. Hutter, “Learning quadrupedal locomotion over challenging terrain”, Science Robotics 5(47), 2020.
• N. Rudin, D. Hoeller, P. Reist and M. Hutter, “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, Conference on Robot Learning (CoRL), 2021.
• A. Kumar, Z. Fu, D. Pathak and J. Malik, “RMA: Rapid Motor Adaptation for Legged Robots”, Robotics: Science and Systems (RSS), 2021.
• L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba and P. Abbeel, “Asymmetric Actor Critic for Image-Based Robot Learning”, Robotics: Science and Systems (RSS), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

This closes the module on learning for manipulation and locomotion. The next module (M10) asks what happens when a policy trained in a simulator meets the real robot: the reality gap, domain randomisation and system identification, stress tests, and the staged rollout of lesson M10-T5.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T5: Curriculum and Task Design

  Order tasks and difficulty so that learning succeeds, and detect when a curriculum hides a failure.

• Lab
  M9-L2: Shape a Gait Reward

  A simulated walker shuffles, hops or falls depending on the reward weights: choose weights that walk.