M6.2 — Behavior Cloning and Imitation Learning

Learning objectives

After this lesson, you should be able to:

• Define imitation learning and behavior cloning as supervised learning on observation-action pairs.
• Clone a simple expert and check the result by rolling it out.
• Explain compounding errors and why a clone fails outside the demonstrated states.
• Describe how DAgger addresses this.
• Explain why squared error averages multimodal demonstrations into a bad action.
• List what these problems imply for training and evaluating a VLA.

The one-sentence idea

Behavior cloning trains a policy to predict the demonstrator’s action from the observation, which works where the demonstrations cover, and fails where errors carry the robot outside them.

The clone only knows the states the expert showed it.

Learning to act by watching

Imitation learning trains a robot to act by copying an expert, usually a person who demonstrates the task. It sidesteps a hard problem: describing in advance, with a formula, what “good” behaviour is. You just show it. The simplest and most widely used form is behavior cloning (BC), which treats imitation as the supervised learning of M6.1:

• Inputs: the observations the expert had (camera images, joint readings, and for a VLA the instruction).
• Labels: the actions the expert took at those moments.
• Model: a policy π(o) → a, trained to predict the expert’s action from the observation.
• Loss: a distance between the predicted and demonstrated action, such as squared error.

This is exactly how a VLA is trained on robot demonstrations, with a very large model and a very large number of examples. Understanding where behavior cloning works, and where it breaks, explains most of the design choices in the later phases of the roadmap.

Behavior cloning in a toy world

Take a one-dimensional “robot” whose position s changes by its action each step, and an expert that always pushes it toward zero, a = −k·s. We record demonstrations as pairs of (state, action), then fit a policy to them by least squares (M6.1):

import numpy as np

rng = np.random.default_rng(0)
k_expert = 0.5
s = rng.uniform(-1, 1, size=100)             # states visited in the demonstrations
a = -k_expert * s                            # the expert's actions

k_learned = -np.sum(s * a) / np.sum(s * s)   # least-squares fit of a = -k s
print(round(k_learned, 3))

state = 0.9                                  # roll out the cloned policy
for step in range(6):
    state = state + (-k_learned * state)
print(round(state, 3))

0.5
0.014

With clean demonstrations and a model that matches the expert’s form, the clone recovers the expert’s gain exactly and drives the state to zero (0.9 → 0.014 in six steps). Real tasks are not like this: the expert is a person, whose actions depend on things the robot cannot see, and the model must learn the mapping from images, which is far harder. What stays the same is the recipe, and the weak points that this toy case hides.

The central problem: compounding errors

Behavior cloning learns from states the expert visited. But once the robot runs its own policy, it makes small mistakes and drifts into states the expert never visited, where the clone has no data and may act badly, which pushes it even further away. Errors compound. Here is the effect in a world where being off-course is unstable: the state grows by 20% each step unless the action cancels it. The expert always cancels it exactly, so every demonstration stays within 0.1 of zero. The clone is a nearest-neighbour policy, which copies the action of the closest demonstration state:

growth = 1.2                                          # unstable: state grows 20% per step if the action is 0
expert = lambda s: -growth * s                        # the expert cancels the growth exactly

demo_s = np.linspace(-0.1, 0.1, 41)                   # demonstrations only ever stay near zero
demo_a = expert(demo_s)

def clone(s, states=None, actions=None):
    states = demo_s if states is None else states
    actions = demo_a if actions is None else actions
    return actions[np.argmin(np.abs(states - s))]     # action of the nearest demonstration state

def rollout(policy, s, steps=15):
    for _ in range(steps):
        s = growth * s + policy(s)
    return s

for start in (0.05, 0.5, 0.8):
    print(start, round(abs(rollout(clone, start)), 3), round(abs(rollout(expert, start)), 3))

0.05 0.0 0.0
0.5 0.003 0.0
0.8 3.681 0.0

The columns are the starting state, the clone’s final distance from zero, and the expert’s. From a start inside the demonstrations the clone does fine, and from 0.5 it recovers, slowly. From 0.8, far outside, it can apply only the largest action it ever saw, which is too small to cancel the growth, so the state grows (to 3.7 after 15 steps, and it keeps growing) while the expert still gets to zero. The robot did not misbehave randomly: it simply faced states its training data never covered.

A standard remedy is DAgger (dataset aggregation). Let the learner drive, record the states it visits, ask the expert what it would have done in each one, add those labelled states to the data, and retrain. The training data now includes the learner’s own mistakes, with the corrections:

states, actions = demo_s.copy(), demo_a.copy()
for round_ in range(3):
    policy = lambda s: clone(s, states, actions)
    print(round_, round(abs(rollout(policy, 0.8)), 3))      # how the policy does before this round's data
    visited, s = [], 0.8
    for _ in range(15):
        visited.append(s)
        s = growth * s + policy(s)
        if abs(s) > 3: break                          # stop a run that is clearly diverging
    states = np.append(states, visited)
    actions = np.append(actions, [expert(v) for v in visited])     # the expert labels them
print("after", round(abs(rollout(lambda s: clone(s, states, actions), 0.8)), 3))

0 3.681
1 0.0
2 0.0
after 0.0

Round 0 uses only the original demonstrations and ends at 3.681. After one round of aggregating the learner’s visited states with the expert’s labels, the clone reaches zero from the same start, and stays there. DAgger needs an expert who can be queried on demand, which is easy for a scripted controller but costly for a person. That is one reason offline demonstration datasets, and techniques that make the policy robust to small deviations, are so central in practice.

When there is more than one right answer

A second problem is that different demonstrators, or the same one on different days, may do different things from the same observation. Suppose a robot meets an obstacle directly ahead, and half the demonstrations swerve left (action −1) and half swerve right (+1). Both are good. A policy trained with squared error is pulled to the average:

demos = np.array([-1.0] * 50 + [1.0] * 50)       # same observation, two equally good actions
best_constant = demos.mean()                      # the squared-error-optimal single prediction
print(best_constant)
print(np.mean((demos - best_constant) ** 2), np.mean((demos - 1.0) ** 2))

0.0
1.0 2.0

The best single prediction under squared error is 0, going straight into the obstacle, which neither demonstrator did. Committing to one option (always +1) costs a higher average error (2.0 against 1.0), so the loss prefers the average, even though it is the one action guaranteed to be wrong. Fixing this means a policy that can represent several possible actions and pick one, rather than output a single number. That is the motivation for discretised action tokens and for diffusion and flow-matching policies, which Phase 4 covers.

What this means for training a VLA

• Data coverage matters most. A clone is only reliable where demonstrations exist. Demonstrations that include recoveries, with the robot ending up off-course and then getting back, teach the policy what to do after a mistake.
• Scale helps coverage. Large and varied datasets, and models pretrained on broad data (M3.5), widen the range of situations a policy handles. Phase 5 covers such data collections.
• Action prediction needs a good output form. Single-number outputs average different good behaviours. Richer outputs, and predicting a short chunk of future actions at once, reduce the problem.
• Evaluate by running the policy. Low prediction error on held-out demonstrations (M6.4) does not guarantee that a robot succeeds when its own errors compound, so success has to be measured by executing the policy, as Phase 8 describes.

Imitation learning has relatives. Inverse reinforcement learning infers the reward the expert was optimising, and reinforcement learning (M6.6) learns from a reward directly. Many practical systems start from behavior cloning and then improve with reinforcement learning.

Common misconceptions

Engineering takeaways

1. Behavior cloning is supervised learning from observations to demonstrated actions.
2. It trains on the expert’s states, but runs on the learner’s own.
3. Small errors push the robot into states the demonstrations did not cover, and errors compound.
4. DAgger adds the learner’s visited states, labelled by the expert, to the data.
5. Squared error averages different valid actions, and the average can be invalid.
6. Richer outputs, such as sampled or diffusion-based actions and action chunks, address multimodality.
7. Demonstrations that include recoveries improve robustness.
8. Evaluate a policy by executing it, not only by prediction error.

A policy imitates well where it has seen the expert, and has to guess everywhere else.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You know how behavior cloning works and where it breaks. Next is what a demonstration dataset actually contains and how it is organised.

Back to the VLA roadmap prerequisites