11.1 — Reinforcement Learning for Robot Policies

Learning objectives

After this lesson, you should be able to:

• Explain the try-compare-adjust loop of a policy gradient.
• Explain why PPO clips the probability ratio.
• Give an example of reward hacking.
• Explain why imitation is often followed by RL.

The one-sentence idea

Improve by trying, in small safe steps, against a reward that means what you want.

The reward is a proxy; optimisation exploits proxies.

Improving a policy by trying things

Imitation (lesson 3.3) can only be as good as its demonstrator, and it does not learn from its own mistakes. Reinforcement learning (RL) learns by trial and reward: the agent acts, receives a reward, and adjusts its behaviour to get more of it. The standard textbook frames this as an agent interacting with an uncertain environment to maximise the total reward it receives.^[1] For robots the appeal is that RL can go beyond the demonstrations and optimise directly for success, and the cost is that trials on real robots are slow, risky and expensive. Legged-robot work illustrates both sides: RL has mostly stayed in simulation because training on hardware is costly and slow, so controllers are trained in simulation and transferred.^[2][3]

The common recipe for VLAs is therefore a combination: first imitation to get a decent policy, then RL to improve it, with care about the risks of exploring.

The policy gradient in 20 lines

The simplest RL method for continuous actions makes the policy random (a Gaussian around its preferred action), tries it, and nudges the policy parameters toward the actions that earned above-average reward. The reaching task of earlier lessons will do. The policy moves a fraction θ of the way to the goal at each step. Imitation left it timid (θ = 0.10) because the demonstrator was cautious; reward is the negative final distance to the goal:

import numpy as np
rng = np.random.default_rng(0)

def episode(theta, noise=0.05, steps=8):
    pos, noise_log = 0.0, 0.0
    for _ in range(steps):
        eps = rng.normal(0, noise)
        pos += theta * (1.0 - pos) + eps                  # the action is theta times the distance to the goal, plus exploration noise
        noise_log += eps
    return -abs(1.0 - pos), noise_log                     # reward, and the total exploration noise used

theta = 0.10                                              # from imitation: moves only 10% of the way each step
print("start: theta", round(theta, 3), "| average reward", round(float(np.mean([episode(theta)[0] for _ in range(500)])), 3))
for update in range(1, 201):
    batch = [episode(theta) for _ in range(64)]
    rewards = np.array([r for r, n in batch]); noises = np.array([n for r, n in batch])
    advantage = rewards - rewards.mean()                  # better than average -> push toward that noise
    theta += 0.2 * float(np.mean(advantage * noises))      # move theta toward the exploration noise that did better
    if update in (50, 100, 200):
        print(f"update {update:>3}: theta {theta:.3f} | average reward {np.mean([episode(theta)[0] for _ in range(500)]):.3f}")

start: theta 0.1 | average reward -0.435
update  50: theta 0.211 | average reward -0.151
update 100: theta 0.283 | average reward -0.082
update 200: theta 0.355 | average reward -0.058

Toy model. A one-parameter policy and a crude gradient estimate (the noise direction weighted by advantage). Real methods use neural networks and more careful estimators, but the loop is the same: try, compare with average, move toward what did better.

The policy parameter grows toward the value that reaches the goal and the reward rises. Notice what drives it: only a number at the end of each try. No demonstration was needed after the starting point.

Why updates are kept small: the idea behind PPO

A big update can wreck a policy that was working. Proximal policy optimisation (PPO) is a widely used method that alternates between collecting experience and optimising a surrogate objective, with an objective that allows several passes over each batch of data instead of a single update per sample; it was designed to keep some of the benefits of the earlier TRPO method while being easier to implement.^[4] The key trick, in the form I know from the literature (the paper itself was not read for this course, so check it), is to clip the probability ratio between the new and old policy so that no single sample can push the policy far. The clipped objective for one sample with advantage A and ratio r is min(r·A, clip(r, 1−ε, 1+ε)·A). What that does:

import numpy as np

eps = 0.2
def clipped(r, A):
    return min(r * A, float(np.clip(r, 1 - eps, 1 + eps)) * A)

print("ratio r = new probability / old probability")
for A, label in ((+1.0, "good action (A = +1)"), (-1.0, "bad action (A = -1)")):
    row = "  ".join(f"r={r:.1f}: {clipped(r, A):+.2f}" for r in (0.5, 0.8, 1.0, 1.2, 1.5))
    print(f"{label:<22} {row}")

ratio r = new probability / old probability
good action (A = +1)   r=0.5: +0.50  r=0.8: +0.80  r=1.0: +1.00  r=1.2: +1.20  r=1.5: +1.20
bad action (A = -1)    r=0.5: -0.80  r=0.8: -0.80  r=1.0: -1.00  r=1.2: -1.20  r=1.5: -1.50

For a good action, the objective stops rewarding increases in probability beyond r = 1.2: nothing is gained by pushing further. For a bad action, it stops rewarding decreases beyond r = 0.8. Each step therefore stays close to the old policy, which is what a robot controller you do not want to break needs.

The catch: you get what you reward

A reward is a proxy for what you want, and optimisation is good at exploiting the gap. Amodei and colleagues list avoiding side effects and avoiding reward hacking among their concrete problems in AI safety, alongside safe exploration and robustness to distributional shift.^[5] A small illustration: you want the hand to reach the goal quickly, so you reward speed. The optimiser finds that going fast through the goal and beyond earns more than stopping at it:

import numpy as np

def run(gain, steps=10, goal=1.0):
    pos, speed_reward, error = 0.0, 0.0, None
    for _ in range(steps):
        move = gain * (goal - pos)
        pos += move
        speed_reward += abs(move)                         # the reward counts how far the hand moves each step
    return speed_reward, abs(goal - pos)

for gain in (0.5, 1.0, 1.9, 2.5):
    reward, final_error = run(gain)
    print(f"gain {gain}: reward for speed {reward:7.2f} | final distance from the goal {final_error:8.3f}")

gain 0.5: reward for speed    1.00 | final distance from the goal    0.001
gain 1.0: reward for speed    1.00 | final distance from the goal    0.000
gain 1.9: reward for speed   12.38 | final distance from the goal    0.349
gain 2.5: reward for speed  283.33 | final distance from the goal   57.665

The highest “speed reward” belongs to the gain that overshoots wildly and never settles. In this toy the failure is obvious; with learned policies and complicated rewards it can hide. Write rewards that include the success condition itself, penalise unsafe behaviour, test the final policy with the evaluation methods of phase 8, and never explore on hardware without the safety layers of phase 9.

Common misconceptions

Engineering takeaways

1. Try, compare with average, adjust.
2. Start from imitation.
3. Clip updates.
4. Test for reward hacking.

Reward what you mean, then check what you got.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: getting good behaviour from less robot data.

Back to the VLA roadmap