M3-T4 — DAgger and Interactive Imitation

Learning objectives

After this lesson, you should be able to:

• Write the DAgger loop, including the mixing parameter beta, and say which data each round adds.
• State the guarantee of Ross, Gordon and Bagnell (2011) in words, with the assumptions it needs.
• Read a DAgger run on Scout: success and dataset size round by round, against behaviour cloning on the same amount of data.
• List the costs of interactive imitation and name three variants that reduce them.

The intuition

Learning to drive from a passenger seat shows the problem. You watch an expert for a month and never once see what a drift into the verge looks like, so when you first drift you have no idea what to do. The better way to learn is to drive while the instructor watches and says what you should have done each time you wander off line. The instructor’s remarks come at your mistakes, which are exactly the situations you will be in again.

DAgger (“dataset aggregation”) is that instructor. Behaviour cloning collects the states the expert visits and learns the expert’s action there. The learner then acts in a different distribution, because its own errors move it. DAgger lets the learner drive, records the states it visits, and asks the expert for the action at each. The expert’s labels are always right; only the states change. Over a few rounds the dataset comes to cover the places the learner really goes, including the bad ones, each labelled with the way back.

The mathematics

The loop

Let the expert be a policy π*, and let the learner be chosen from a class of policies by minimising a supervised loss ℓ between its action and the expert’s. Write D for the dataset of (state, expert action) pairs.

1. Round 0: collect expert demonstrations and set D to their (state, expert action) pairs. Train the first learner π̂₁ on D.
2. In round i, build a driving policy π_i = β_i · π* + (1 − β_i) · π̂_i. With probability β_i the expert’s action is executed at a step, otherwise the learner’s. Run it for several episodes and record the states visited.
3. Ask the expert for the action π*(s) at every recorded state s, whichever policy drove. The executed action does not matter; the label is the expert’s.
4. Aggregate: D ← D ∪ {(s, π*(s))}. Train π̂_{i+1} on all of D, not only the new pairs.
5. Repeat for N rounds and keep the best policy of the sequence by validation on fresh episodes (lesson M2-T6).

Ross, Gordon and Bagnell (2011) use β₁ = 1, so round 1 is plain cloning, and let β decay quickly, for example geometrically or to zero after the first round. A positive β early keeps the learner from wandering into very bad states while it is still poor; at β = 0 the learner drives alone.

π*

the expert policy, which can be queried at any state

π̂_i

the learner trained at round i

β_i

probability of executing the expert’s action at a step in round i

D

aggregate dataset of (state, expert action) pairs

T

episode horizon, in steps

ε

per-step probability of a mistake (or per-step loss) on the data the learner is trained on

Why the drift is linear, not quadratic

In behaviour cloning the loss is small on the expert’s state distribution, but the policy is deployed on its own. After one mistake at step t the learner may be in a state outside the data, where it can go on making mistakes for the remaining T − t steps, so the cost of an early mistake is of order T. With about εT early mistakes, each costing up to T, the total is of order ε T² (lesson M3-T3).

DAgger trains the learner on the distribution of states induced by learner policies. Learning on a distribution that changes as the learner changes is an online learning problem, and a no-regret online learner (for example, follow-the-leader on a strongly convex loss, which is what “retrain on all of D” is) has average loss that approaches the loss of the best policy in hindsight. Ross et al. show that, with a number of rounds that grows roughly linearly in T (up to log factors), some policy in the sequence has cost within about T·ε_N of the expert’s, where ε_N is the loss of the best policy in the class on the aggregate data, plus a lower-order term. The error is linear in T, not quadratic.

Assumptions you must keep in mind

• A queryable expert. The expert must give a correct action at any state the learner reaches, including states it would never reach itself. A human can do this only if the interface lets them see the state and pick an action there.
• A small ε_N. The guarantee is relative to the best policy in the class on the aggregate data. If the class cannot imitate the expert on the visited states (no pillar information in the features, say), ε_N is large and the bound says nothing useful.
• Consistent loss and a no-regret learner. The proof uses a bounded loss and a learner whose average regret goes to zero; practical neural-network training only approximates this.
• Enough rounds and data. The bound has a term that shrinks as the number of rounds grows; a handful of rounds can be enough in practice, but nothing in the theory promises it.
• A stationary world. The states, the expert and the dynamics must not change from round to round.

The quadratic bound for cloning is also a worst case. If the expert can recover from a mistake at constant cost, however late it happens, cloning behaves better than T² (the original paper expresses this through a bound on how much worse than the expert a single deviation can be). Scout’s hall is like that away from the pillar and unlike it in front of it.

The cost

A worked example

All numbers come from running RLCore in node on Scout’s world. The learner is a table clone: Scout’s hall is cut into cells of 1 m, each cell stores the mean expert action of the data that fell in it, and an empty cell stores the action (0, 0). It is deployed with tablePolicy. It is a deliberately coarse learner: it cannot guess an action in a cell with no data, which is exactly what reveals a coverage gap. Each policy is scored on 100 starts from seed 500 in two sets: 'all' starts and 'near' starts (close to the line through the pillar).

The run. Round 0 is 8 expert demonstrations from starts(11, 8, 'all'), 268 (observation, action) pairs. In each later round the current learner drives 10 new starts from starts(11 + 100 r, 10, 'all') with β = 0, every state it visits is labelled by the expert, and the table is rebuilt from all data so far.

Every failure in this table is a timeout, not a crash: in all six rows the crash count is 0 on both sets. The coarse table has no data for the cells the learner reaches, stores (0, 0) there, and the robot stops. That is the compounding error of lesson M3-T3 in its simplest form: the first error puts Scout in a cell with no data, and there the policy cannot recover. Round 1 shows it directly. The ten starts produce 745 new pairs, almost three times the demonstrations, because stalled episodes run the full 80 steps and every step of a stall is a labelled state. The expert’s label for those states is the way forward, so the next table has an entry there.

Same amount of data, expert states only. Behaviour cloning on the same number of pairs (demonstrations from starts(11, n, 'all') with n chosen to give at least as many pairs):

More expert demonstrations barely help: 93 demonstrations fill 37 cells, 6 fewer than the DAgger dataset’s 43, and the clone still succeeds from 29 starts out of 100 and from none of the 100 near starts. The expert’s paths cover a thin band through the hall; the learner’s own, mistaken paths cover the places it needs advice. Adding data is not the same as adding the right data.

Is it one lucky run? The same experiment (8 demonstrations, 5 rounds of 10 starts, β = 0) was repeated for 10 seeds, 21 to 30. The mean success went from 10.2 to 78.9 out of 100 on all starts and from 2.9 to 88.9 on near starts. Final success on all starts ranged from 42 to 95, so the shape (large rise, then a plateau that depends on the seed) is consistent but the exact value is not.

The mixing parameter. Repeating the first run with β_i = 0.5^i (the expert’s action is executed at a step with probability 0.5, 0.25, 0.125, …) gives 16, 26, 24, 66, 95, 95 successes on all starts across rounds 0 to 5. It ends in the same place, with fewer new pairs per round early (315 against 745 in round 1). It is worse in rounds 1 and 2 (26 and 24 against 54 and 61) and level or ahead from round 3 (66 against 64, then 95 against 86), which one seed cannot rank. In this world there is nothing dangerous to protect the learner from, so the benefit of β is not visible; on a real robot it is the safety.

What the simulation simplifies. The expert is a deterministic function that answers instantly and exactly at any state, which is the best case for DAgger. A human expert is slow, noisy and inconsistent, and often cannot say what they would do in a state they would never be in. The coarse table is chosen for clarity; a k-nearest-neighbour clone with 15 expert demonstrations already succeeds from 100 of 100 starts in the same hall, so there DAgger has little to fix. How much DAgger helps depends on how far the learner’s errors take it from the data, not on the algorithm alone.

Engineering implications and limitations

Variants that reduce the cost

These descriptions are at the level of the idea; read the papers for the algorithms.

• Human-gated DAgger (HG-DAgger; Kelly et al., 2019). The learner drives and the human expert watches and takes over when they judge the situation is going wrong. Labels are recorded only while the human is in control, so the human labels by acting, not by reviewing frames, and the learner’s early mistakes are cut short. The cost is that the human’s decision to intervene is itself a signal, and the data it gives is biased towards the moments someone chose to step in.
• Intervention learning. The family of methods that treat an intervention as the training signal: the human’s corrective actions are added to the data, and the moment of intervening can also indicate that the preceding states were bad. Methods differ in how they weight the corrections and what they do with the states before the intervention.
• DART (Laskey et al., 2017). Instead of putting the learner in the loop, the supervisor’s demonstrations are collected while noise is injected into the supervisor’s actions during execution, so that the demonstrations include recoveries from perturbed states without the expert ever being replaced. The noise level is tuned to match the learner’s expected error. The labels are still the supervisor’s actions, not the noisy executed ones.
• Gated and confidence-based variants. Ask for labels only where the learner is uncertain, or where it disagrees with a safety monitor, to save expert time. How well uncertainty tracks the states that need labels is an empirical question.

Practical advice

• Label by acting if you can. Frame-by-frame labelling of a recorded learner run produces actions the person would not take in real time. Interventions on a live learner give better labels.
• Keep the first rounds safe. Start with β above zero, limit speed, and bound the workspace. Decide who stops the robot and how before round 1.
• Version everything. Keep the dataset, the labels, the policy and the evaluation episodes of each round, so a round that made things worse can be removed (lab M3-L3, add-dagger-corrections, does this with a log).
• Evaluate on starts not used in any round. The rounds’ starts are training data now (lessons M1-T5 and M2-T6). Held-out evaluation starts must be kept apart.
• Weight or trim stalls. A stalled robot generates many near-identical states; keep them, but do not let one dull state dominate the loss.
• Know when not to use it. If the expert cannot answer at the learner’s states, or a wrong learner is dangerous, clone more carefully and collect broader demonstrations (lesson M3-T5) instead.

Common misconceptions

Engineering takeaways

1. DAgger runs the learner, has the expert label the states the learner visits, aggregates all data and retrains; β mixes expert and learner actions while driving.
2. The data then matches the states the policy will meet, and the error in deployment grows linearly in the horizon, under assumptions: a queryable expert, a small best-in-class loss and a no-regret learner (Ross, Gordon and Bagnell, 2011).
3. On Scout, a coarse table clone rose from 16 to 95 successes in 5 rounds, while behaviour cloning with the same amount of data reached 29.
4. The price is an expert in the loop, a labelling burden, and safety while a poor learner drives.
5. HG-DAgger, intervention learning and DART reduce those costs, each by moving the expert’s effort to different moments.

References and further reading

The DAgger paper and two of its practical descendants.

• S. Ross, G. Gordon and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, AISTATS, 2011.
• M. Laskey, J. Lee, R. Fox, A. Dragan and K. Goldberg, “DART: Noise Injection for Robust Imitation Learning”, CoRL, 2017.
• M. Kelly, C. Sidrane, K. Driggs-Campbell and M. J. Kochenderfer, “HG-DAgger: Interactive Imitation Learning with Human Experts”, ICRA, 2019.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: DAgger only helps as much as the labels are good, and so does plain cloning. The following lesson looks at the demonstrations themselves: coverage, consistency, noise and how many you need.

Related content

Nothing below is required to finish this lesson.

• Before
  M3-T3: Distribution Shift and Compounding Errors

  Explain why small errors take a cloned policy to states it never saw, and why the error grows with the horizon.

• Lab
  M3-L3: Add DAgger Corrections

  Run the clone, ask the expert what it should have done at the states it reached, and retrain until the drift stops.