3.3 — Behavior Cloning and Imitation Learning

Learning objectives

After this lesson, you should be able to:

• Explain behavior cloning as supervised learning on (observation, action) pairs.
• Explain why a cloned policy drifts into situations not in the data.
• Describe how adding expert labels on the policy’s own states helps.
• Name the cost of that fix.

The one-sentence idea

Behavior cloning copies the demonstrator, and fails where the demonstrator never went.

The policy’s own mistakes change what data it sees next.

Learning by copying

Behavior cloning (BC) is the simplest way to teach a robot: show it what a good demonstrator did and train it to do the same. Technically it is plain supervised learning. The input is the observation, the label is the demonstrator’s action, and the model is trained to predict the label. An early example is ALVINN, a three-layer neural network that took camera and laser range finder images and output the direction the vehicle should travel to follow the road.^[1] Modern VLAs follow the same pattern at a much larger scale: OpenVLA, for example, is trained with next-token prediction on action tokens from its demonstration data.^[2]

The next example is the whole idea in a few lines. A demonstrator moves a hand toward a goal with a simple rule, and a linear model is fitted to its (observation, action) pairs. Then the learned policy is run on its own:

import numpy as np
rng = np.random.default_rng(0)

def demonstrator(hand, goal):
    return 0.3 * (goal - hand)                       # the hidden rule: move 30% of the way to the goal

# 1. Record demonstrations
X, Y = [], []
for _ in range(100):
    goal = rng.uniform(-1, 1, 2); hand = rng.uniform(-1, 1, 2)
    for _ in range(10):
        a = demonstrator(hand, goal) + rng.normal(0, 0.005, 2)      # a slightly imperfect human
        X.append(goal - hand); Y.append(a)                          # observation feature -> action
        hand = hand + a
X, Y = np.array(X), np.array(Y)

# 2. Train: ordinary least squares
W, *_ = np.linalg.lstsq(X, Y, rcond=None)
print("learned gain:", np.round(np.diag(W), 3).tolist(), "(the demonstrator used 0.3)")

# 3. Run the learned policy on its own
hand, goal = np.array([-0.8, 0.9]), np.array([0.7, -0.4])
for _ in range(15):
    hand = hand + (goal - hand) @ W
print("distance to goal after 15 steps:", round(float(np.linalg.norm(goal - hand)), 4))

learned gain: [0.3, 0.3] (the demonstrator used 0.3)
distance to goal after 15 steps: 0.0094

Toy model. A made-up demonstrator with one hidden rule, so that a linear model can recover it. Real policies are deep networks fitted to messy human data.

The model recovered the rule and the robot reaches the goal. When it works, BC is as easy as that. The trouble comes from what the demonstrations did not include.

Why copying breaks down: the policy visits places the demonstrator never did

A demonstrator is good, so the demonstrations stay in good situations. A learned policy makes small mistakes, a mistake takes it to a situation the demonstrations never covered, where it is more likely to be wrong, and the errors pile up. This is the central problem of imitation learning. Ross, Gordon and Bagnell put it as follows: sequential problems like imitation learning break the i.i.d. assumption of standard statistical learning, which hurts performance.^[3]

Here is that effect in a small experiment. A car drifts away from the lane centre unless it steers back, and the drift gets worse the farther away it is (an unstable system, like balancing). The expert steers back firmly. We record expert driving, in which the car stays within about ±0.1 of the centre, and train a “nearest recorded example” policy on it. Then both drive with random pushes:

import numpy as np
rng = np.random.default_rng(0)

def expert(e):                         # steer back toward the lane centre
    return -1.8 * e

def drive(policy, noise, steps=40):
    e, seen = 0.0, []
    for _ in range(steps):
        seen.append(e)
        e = 1.5 * e + policy(e) + rng.normal(0, noise)     # unstable: the error grows by itself unless corrected
        if abs(e) > 1.0:                                    # left the road
            return seen, True
    return seen, False

def nearest_example_policy(states, actions):
    return lambda e: actions[np.argmin(np.abs(states - e))]

def failure_rate(policy, trials=200, noise=0.12):
    return float(np.mean([drive(policy, noise)[1] for _ in range(trials)]))

# Record expert demonstrations (small pushes, so the car stays near the centre)
states = np.concatenate([drive(expert, 0.03)[0] for _ in range(30)])
actions = expert(states)
print("demonstrated states range from", round(float(states.min()), 2), "to", round(float(states.max()), 2))

cloned = nearest_example_policy(states, actions)
print("expert, driving with strong pushes: leaves road in", failure_rate(expert), "of runs")
print("cloned, same pushes:                leaves road in", failure_rate(cloned), "of runs")

demonstrated states range from -0.12 to 0.1
expert, driving with strong pushes: leaves road in 0.0 of runs
cloned, same pushes:                leaves road in 0.305 of runs

Toy model. A one-number “car” with unstable dynamics, chosen to make the effect visible. The size of the failure rate is a property of this toy, not of real robots.

With the same pushes the expert never leaves the road, but the clone does in roughly three runs in ten. Whenever a push moves the car beyond ±0.1, the clone has never seen such a situation, so it copies the nearest thing it knows, which steers too weakly, and the error keeps growing.

The fix: ask the expert about the places you actually go

The same paper proposes an iterative method (nowadays called DAgger) that trains a single deterministic policy and obtains a policy that performs well on the state distribution that the policy itself induces.^[3] In words: let the current policy drive, record the states it really visits, ask the expert what it would have done there, add those labelled examples to the dataset and retrain. The next example adds one such round to the one above:

import numpy as np
rng = np.random.default_rng(0)

def expert(e): return -1.8 * e

def drive(policy, noise, steps=40):
    e, seen = 0.0, []
    for _ in range(steps):
        seen.append(e)
        e = 1.5 * e + policy(e) + rng.normal(0, noise)
        if abs(e) > 1.0:
            return seen, True
    return seen, False

def nearest_example_policy(states, actions):
    return lambda e: actions[np.argmin(np.abs(states - e))]

def failure_rate(policy, trials=200, noise=0.12):
    return float(np.mean([drive(policy, noise)[1] for _ in range(trials)]))

states = np.concatenate([drive(expert, 0.03)[0] for _ in range(30)])
actions = expert(states)
policy = nearest_example_policy(states, actions)
print("behavior cloning only:", failure_rate(policy))

# let the clone drive with pushes, record where it goes, and label those states with the expert
visited = np.concatenate([drive(policy, 0.12)[0] for _ in range(30)])
states = np.concatenate([states, visited]); actions = np.concatenate([actions, expert(visited)])
policy = nearest_example_policy(states, actions)
print("after one round of corrections:", failure_rate(policy))

behavior cloning only: 0.285
after one round of corrections: 0.0

The failures disappear, because the dataset now contains the situations the policy gets itself into, labelled with what to do there. The catch is cost: someone has to provide expert labels while the robot drives, which is slow with real robots. Other approaches reduce the damage differently. For example, ACT predicts chunks of actions and uses temporal ensembling to reduce compounding errors,^[4] the idea we met in lesson 1.5.

Common misconceptions

Engineering takeaways

1. BC = supervised learning on demonstrations.
2. The policy’s mistakes lead to unseen states.
3. Expert labels on the policy’s own states fix this at a cost.
4. Chunking is another way to reduce compounding error.

A copy is only as good as the situations it has seen.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: what the model is optimised for, and how to tell whether it actually works.

Back to the VLA roadmap