Skip to the lesson
FixTheRobot Explore Robotics / Robot Learning / M3 Imitation Learning / M3-T2

Robot Learning · Theory · M3

M3-T2 — Behavior Cloning

Behavior cloning fits a function from observations to the expert’s actions, and its training error says little about how the robot behaves when the function drives.

  • Lesson 2 of 6
  • Module 3: Imitation Learning
  • Theory
  • About 13 minutes to read

Behavior cloning (BC) is the simplest imitation method and the base of most others: treat the demonstrations as a labelled dataset, train a model to predict the expert’s action from the observation, and run the model as the policy. It reuses everything from module M2. This lesson writes the loss, compares policy classes, reproduces the finding from modules M1 and M2 that the features decide what a linear clone can represent, and shows the central warning of the module: a clone can have a small training error and still fail when it is let drive.

Learning objectives

After this lesson, you should be able to:

  • Write the behavior-cloning objective as supervised learning with a squared-error or cross-entropy loss.
  • Compare linear, nearest-neighbour and neural policy classes for cloning and say what each assumes.
  • Pair observations with actions correctly and say what action chunking and observation history add.
  • Explain, with numbers, why training error and closed-loop success can disagree.

The intuition

Behavior cloning asks a policy to answer one question, over and over: given what I see, what would the expert do? The training set is a stack of examples of exactly that, and any regression or classification method can be fitted to it. Nothing about the robot appears in the algorithm, which is why it is fast, simple and easy to run offline on a laptop.

What makes it more than a regression exercise is where the answers are used. The model’s output is not scored on a held-out table; it is sent to the motors, and the next observation depends on it. Two clones with the same training error can behave very differently once the loop is closed, and the figure shows it: the training error falls steadily down the list, but what the robot does depends mainly on whether the model can represent the expert’s answers where it will go.

Training error and closed-loop success of five behaviour-cloning policiesGrouped bars for five clones: success out of 100 on all starts and on starts near the pillar line. Raw linear 0 and 0, training MSE 0.0745. Direction linear 45 and 0, MSE 0.0593. Pillar linear 100 and 100, MSE 0.0088. kNN k=1 table 100 and 100, MSE 0.0084. kNN k=5 table 100 and 100, MSE 0.0066. 050100 00Raw linearMSE 0.0745450Direction linearMSE 0.0593100100Pillar linearMSE 0.0088100100kNN k=1 tableMSE 0.0084100100kNN k=5 tableMSE 0.0066 all startsstarts near the pillar line success / 100
Five clones fitted to the same 945 pairs. Training MSE falls from left to right; success is 0, 45 and then 100 of 100. The two numbers per clone are the success on all starts and on starts near the pillar line, 100 held-out starts each.

The mathematics

The objective

Given a dataset D of N pairs, choose parameters θ of a policy πθ so that its action matches the expert’s on those observations.

continuous actions (squared error)
L(θ)=(1N)⋅∑i ∣∣πθ(oi)−ai ∣∣2L(\theta )=(\frac{1}{N})\cdot \sum _{i}\,\lvert \rvert \pi _{\theta }(o_{i})-a_{i}\,\lvert \rvert ^{2}
discrete actions (cross-entropy over K choices)
L(θ)=−(1N)⋅∑i log⁡ πθ(ai ∣oi)L(\theta )=-(\frac{1}{N})\cdot \sum _{i}\,\log \,\pi _{\theta }(a_{i}\,\mid o_{i})
the clone
π^=arg min⁡θ L(θ)\hat{\pi}=\operatorname*{arg\,min}_{\theta }\,L(\theta )
πθ(o)
the policy: a function from observation to action (or to a distribution over actions)
L(θ)
the training loss, a number you can compute without running the robot
N
the number of pairs, 945 for the 30 Scout demonstrations
K
the number of discrete actions, when the expert’s commands are labels such as forward, left, right

The squared-error loss is the negative log-likelihood of a Gaussian with fixed variance around the prediction, so the model is trained to predict the mean of the expert’s action given the observation. That is the right thing if there is one sensible action per observation, and the wrong thing if there are two (go left or right round the pillar): the mean is then straight into the pillar. Lesson M3-T6 treats this multimodal case. Cross-entropy, a mixture or a generative policy handles it; plain MSE does not.

Policy classes

ClassFormWhat it assumesTypical failure
Linear on featuresa = Wᵀ φ(o)The action is a linear function of hand-built features φCannot represent anything the features do not expose (lesson M2-T2)
k-nearest neighboursa = mean of the k closest examplesSimilar observations get similar actions; the distance is meaningfulNo generalisation outside the data: copies the nearest example, even if far away
Small neural networka = fθ(o), a few layersA smooth non-linear function; features are learnedOverfits with little data; extrapolates unpredictably
Large visuomotor network, transformer or diffusion policyImage and proprioception in, an action or chunk outEnough data to learn perception and control togetherNeeds hundreds to thousands of demonstrations; hard to debug

The M1 and M2 finding carries over unchanged: the features decide what a linear clone can represent. With raw [dx, dy] the best linear map cannot produce the expert’s unit-length direction; with the goal direction it can steer at the goal but not round the pillar; only features that expose the pillar (lesson M2-T2) let it. No amount of extra data fixes a missing feature.

Data hygiene

  • Pair ot with at. The label for the observation at step t is the action chosen on it. A label from one step earlier or later trains the model to predict a stale or future command. Check the alignment with a plot before training.
  • Split by episode. Consecutive steps of one demonstration are nearly identical, so a random split of pairs leaks (lesson M2-T6). Hold out whole demonstrations.
  • Normalise. Scale observations (lesson M2-T2) so that distances and gradients behave. For kNN this decides which dimension matters.
  • Clip to the actuator range. Train on the command the robot actually executed (after limits), and clip the policy’s output the same way.

History and chunks

The policy may take more than the latest observation: a short history (ot-h ... ot) helps when the state is not observable from one frame, for example to estimate velocity, but it also lets the policy copy its own recent actions, a trap known as copycat or causal confusion that makes the training error small and the behaviour poor. The policy may also predict a chunk of future actions (at ... at+k-1), executed open-loop or blended. Chunking reduces the number of times the policy can make a fresh error and smooths the motion; it is used in recent imitation systems such as ACT and Diffusion Policy, but it does not remove the distribution shift of lesson M3-T3.

What BC is good at

BC trains offline on logged data with a standard supervised pipeline: no simulator, no exploration on the real robot, no reward. A new demonstration set means a few minutes of fitting. It is the quickest route to a baseline, and the baseline is often strong when demonstrations cover the states the robot will visit.

A worked example

All numbers come from RLCore. The data are the 945 pairs from the 30 demonstrations of lesson M3-T1 (starts(1, 30, 'all')). Every policy is evaluated in closed loop on 100 held-out starts per set from seed 500 (starts(500, 100, 'all') and starts(500, 100, 'near'), where “near” means y between 2 and 4 m, close to the line through the pillar). The expert itself succeeds on 100 of 100 of each set, with mean return 8.46 on “all”.

Linear clones: only the features differ. Ridge regression, no regularisation, on three feature sets:

PolicyTraining MSEHeld-out MSESuccess, allSuccess, nearCrashes, all
Raw [dx, dy, 1]0.07450.07740 / 1000 / 10048
Direction [dx/|d|, dy/|d|, 1]0.05930.061545 / 1000 / 10052
Pillar-aware (7 features)0.00880.0092100 / 100100 / 1000

The held-out MSE is measured on the expert actions of 30 fresh demonstrations from starts(500, 30, 'all'); it is close to the training MSE, so these three clones are not overfitting. They differ in capacity: the raw and direction models underfit. The Wilson 95% interval for 100 of 100 is [0.963, 1.000], and for 45 of 100 it is [0.356, 0.548].

The gap between low training error and success. Compare the direction clone with the raw clone: the MSE falls by 20% (0.0745 to 0.0593) and success on “all” rises from 0 to 45. Compare the direction clone with the pillar clone: the MSE falls by a factor of 6.7 and success doubles. A small difference in MSE hides a qualitative change: the direction clone’s error is concentrated in the few steps next to the pillar, where the expert’s action changes sharply, and there a small error is a crash. Squared error is averaged over all 945 steps, while a crash is decided in the few steps near the pillar.

The same gap with a flexible model. Fit a 5-nearest-neighbour clone (distance on the four raw observation values) on only 5 demonstrations, 156 pairs. Its training MSE is 0.0023, essentially zero, and it succeeds on all 5 of its training starts. On the 100 held-out “all” starts it succeeds on 61, crashes 11 times and times out in 28. A low training error with a high-capacity model says that it memorised the data, not that it behaves like the expert.

kNN clones as a policy table. To run a kNN clone from Python in the browser labs, the learner returns an action for the observation at the centre of every 0.5 m cell of the hall and the world looks the nearest cell up (RLCore.gridQueries(0.5) lists the 240 cells, RLCore.tablePolicy(actions, 0.5) is the policy). Build the table with k-nearest neighbours over the 945 pairs:

CloneTraining MSE (table lookup)Held-out MSESuccess, allSuccess, nearSuccess, easy
kNN, k = 10.00840.0081100 / 100100 / 10089 / 100
kNN, k = 50.00660.0066100 / 100100 / 10083 / 100

Both succeed on “all” and “near”, in the middle of the demonstrated region. Two points need honesty. First, the extra column, “easy”, is the set of starts close to the top and bottom walls (starts(500, 100, 'easy')): the table clones fail there, 11 of 100 for k = 1 and 17 for k = 5. Every failure is a time-out at the east wall: the robot ends at x = 10 m, y ≈ 3 m, 1 m past the goal circle, pushed against the wall by the action stored in the last cell. That is not a pillar failure and not a lack of capacity: it is the table’s resolution (0.5 m cells, one action per cell) near a goal that is only 0.5 m wide, in a part of the hall with few examples. The direct kNN policy, which does not quantise, scores 100, 100 and 88 of 100 on the three sets at k = 1 and 96, 98 and 86 at k = 5. The numbers do not say k = 1 is better in general; with 100 starts per set the differences between 83, 86, 88 and 89 are inside the noise (a Wilson interval for 89 of 100 is [0.814, 0.937]).

Second, the training MSE of k = 1 as a table, 0.0084, is not zero even though a 1-NN model reproduces its training labels: the table replaces each cell by the action of one example near its centre, and the policy then looks the cell up for each observation inside it. The deployment form changed the model, and the error moved with it.

A one-step label error. Train the pillar-aware linear clone with labels shifted by one step, pairing ot with the previous action at-1: training MSE 0.0114, success on “all” 92 of 100 with 8 crashes, against 100 of 100 with the correct alignment. Pairing ot with the next action at+1 gave MSE 0.0076 and 100 of 100. The expert’s command changes slowly (the step is 0.25 s) so a shift is cheap here, and a shift in the other direction is not free. On a real robot with latency of several steps or a fast task the effect would be larger; this is a measurement on one simulated expert, not a rule.

What the simulation simplifies. The expert is deterministic, so the labels are noise-free and one action per observation exists; the pillar-aware linear clone can therefore match it almost exactly. A human demonstrator’s labels are noisy and sometimes multimodal, which is what makes MSE fitting harder in practice. The world also has no sensing noise, no latency and a single obstacle.

Engineering implications and limitations

Plot the labels against the observations

Before fitting, plot the expert’s action against the main observation variables. A cloud with two branches (left and right of the pillar) means MSE will average them; a lag between the action and the motion means the labels are misaligned.

Report closed-loop success, not only the loss

Evaluate the clone as lesson M1-T5 prescribes: held-out starts, several seeds, an interval, and the stratum where it is worst. The training loss is for debugging the optimiser. It is not the metric.

Hold out demonstrations

Split by whole demonstration or by day, not by pairs. A model that has seen the neighbouring steps of the same episode will look better than it is.

Keep the model matched to the data

With tens of demonstrations, a low-capacity model with good features is hard to beat and easy to read. With thousands, a network can learn its own features. A kNN or a tiny network is a cheap diagnostic: if neither works, suspect the data or the observations before the model.

Mind the deployment form

The policy that runs on the robot may differ from the one you trained: quantised, clipped, run at another rate, fed delayed observations. Evaluate the deployed form. The table policy above is a small example of the same mistake.

Add safety outside the clone

BC has no notion of a constraint. Clip the output to actuator limits and put a monitor around it (module M9, lesson M10-T5).

Common misconceptions

“A low training loss means a good policy.”

The 5-demonstration 5-NN clone had a training MSE of 0.0023 and succeeded on 61 of 100 held-out starts. The loss is measured on the expert’s states; the robot drives in its own.

“If it fits the expert’s actions, it has the expert’s behaviour.”

Only on average over the data. The direction clone has a modest training MSE of 0.0593 and crashed on all 100 starts near the pillar, because the few steps it gets wrong are the ones that matter.

“BC needs a different algorithm for robots.”

It is ordinary supervised learning. What changes is how it is evaluated (in closed loop) and what the data contain (the expert’s states only).

“Predicting a chunk of actions removes the drift.”

It reduces the number of decisions and smooths the motion; it does not teach the policy what to do in states the expert never visited.

Engineering takeaways

  1. Behavior cloning is supervised learning of a = πθ(o) from expert pairs, with squared error for continuous actions and cross-entropy for discrete ones.
  2. Features decide what a linear clone can represent: on Scout, raw, direction and pillar features gave success of 0, 45 and 100 of 100 with training MSE 0.0745, 0.0593 and 0.0088.
  3. Training error can be tiny and the closed-loop result poor: a 5-NN clone with MSE 0.0023 succeeded on 61 of 100 held-out starts.
  4. Pair ot with at, split by episode, and evaluate the policy in the form that runs on the robot.
  5. BC is fast, offline and safe to train; its weakness is states the demonstrations do not show, the subject of lesson M3-T3.

References and further reading

An early end-to-end driving clone, a recent end-to-end driving example, and sources for chunked and diffusion policies.

  • D. A. Pomerleau, “ALVINN: An Autonomous Land Vehicle in a Neural Network”, Advances in Neural Information Processing Systems 1 (NIPS 1988), 1989.
  • M. Bojarski et al., “End to End Learning for Self-Driving Cars”, arXiv:1604.07316, 2016.
  • T. Z. Zhao, V. Kumar, S. Levine and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware”, Robotics: Science and Systems, 2023.
  • C. Chi et al., “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion”, Robotics: Science and Systems, 2023.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

Write the behaviour-cloning loss for continuous actions, and say what the model learns to predict when the expert sometimes goes left and sometimes right from the same observation.

Explanation

L = (1/N) Σ ||πθ(oi) - ai||². The model learns the conditional mean of the actions, which for two equally likely branches around an obstacle is the straight line into it; this is the multimodal problem of lesson M3-T6.

Question 2

The direction-feature linear clone has a training MSE of 0.0593 against 0.0745 for the raw one, yet it succeeds on 45 of 100 starts where the raw one succeeds on 0. The pillar-aware clone has an MSE of 0.0088. Why does a 20% drop in MSE change success so much, and a further drop of a factor 6.7 change it again?

Explanation

MSE averages over all steps; success is decided in the few steps near the pillar. The direction clone steers at the goal (so it succeeds from starts away from the pillar line) but cannot swerve, so it crashes near that line; only the pillar features expose the pillar. A scalar loss does not map linearly to closed-loop behaviour.

Question 3

A 5-NN clone trained on 5 demonstrations has a training MSE of 0.0023 and 5 of 5 training starts, but 61 of 100 on held-out starts. Which two checks do you run first?

Explanation

Compare the training and held-out errors and scores, split by whole demonstrations, and plot where the failed rollouts are against the visited cells of the dataset. The gap points to coverage and overfitting (memorisation), not an optimiser problem.

What’s next

Next: lesson M3-T3, Distribution Shift and Compounding Errors, explains why a clone with a small per-step error can still drift far from the expert, and measures it.

Practice this lesson

Clone the Expert

Learn to follow the corridor from 40 expert demonstrations and measure how well the clone does on the same route.

Next lesson

M3-T3 — Distribution Shift and Compounding Errors

Explain why small errors take a cloned policy to states it never saw, and why the error grows with the horizon.

Nothing below is required to finish this lesson.

  • Before
    M3-T1: Learning from Demonstrations

    Say what a demonstration contains, who can provide one, and when imitation is the right way to teach a robot.

  • Lab
    M3-L1: Clone the Expert

    Learn to follow the corridor from 40 expert demonstrations and measure how well the clone does on the same route.