M3-T5 — Collecting Good Demonstrations

Learning objectives

After this lesson, you should be able to:

• Name the properties of a good demonstration set: coverage, consistency, quality, diversity, and enough of it.
• Compute three quality metrics from demonstration logs: state coverage, action variance in nearby states and success rate of the demonstrations.
• Read a learning curve of clone success against the number of demonstrations and say what it does and does not tell you.
• Decide what to filter, what to keep, and what to record so a dataset can be audited later.

The intuition

Teaching someone to cook by letting them watch you make the same dish ten times works, until the day the oil is too hot. They have seen you cook only in the right conditions, so they have no idea how to recover. A good teacher does not just repeat the best case; they cook with varied ingredients, show mistakes being fixed, and, if two teachers will both be watched, agree first on how to chop the onions.

A demonstration set is the same. It is the whole of what the learner will know, so its gaps become the policy’s gaps. The learning curve in the figure is the other half of the picture: with few demonstrations the result depends on which demonstrations you happened to get (the bar), and beyond some number it stops improving, because the remaining failures are no longer about quantity.

The mathematics

What makes a set good

Three metrics you can compute without training

Each is computed from the logs alone. None replaces evaluating the policy; they tell you where to look first.

State coverage

Divide the space the robot may be in into cells (Scout’s hall into 0.5 m squares gives 240 cells) and count the cells that contain at least one demonstrated state. A better version asks about states the policy will reach: take a set of reference states, for example those the expert visits from new starts, and compute their mean distance to the nearest demonstrated state.

Action variance in nearby states

Take pairs of states from different episodes that are close, and average the squared difference of their actions. If the demonstrators are consistent, close states have close actions and the number is small. A large value means the demonstrators disagree, the operator is erratic, or the state leaves out something that matters (the pillar’s side, say, or the time).

Success rate of the demonstrations

The fraction of recorded episodes that reach the goal without a crash or a timeout. For human demonstrations this is a quality rate; for a scripted expert it is a check that the script and the world agree.

D

the set of demonstration pairs (state s, action a) with an episode label

r

neighbourhood radius for “nearby states” (0.25 m in the example below)

cell

a square of the grid over the space (0.5 m in the example below)

gap

mean distance from a reference state to the nearest demonstrated state

How many demonstrations?

There is no universal number; it depends on the variety of the task, the learner and the quality. Learning theory offers only the broad picture: error falls as data grows and then flattens at a floor set by the learner and the data’s quality. The practical method is empirical: train on 25%, 50% and 100% of what you have, evaluate on held-out episodes (lesson M2-T6), and look at the slope. If the curve is still rising, collect more; if it is flat, collect different data.

A worked example

All numbers come from running RLCore in node on Scout’s world. The clone is a 5-nearest-neighbour regressor on the four observation values (average of the 5 closest demonstrated actions). Unless stated, each result is the mean over several demonstration sets with different starts and is scored on 100 starts from seed 500 ('all', and 'near' = close to the pillar line). The demonstrator is the scripted expert, so “noise” and “mistakes” below are added by wrapping the expert, as a simple model of a human.

1. Learning curve

The curve rises steeply to 20 demonstrations and is flat after 40; the spread between sets, 47 points at 5, is gone at 80. Coverage and gap improve in step with success: the gap falls from 0.17 m to 0.026 m. For a learner that could not generalise (a table of 1 m cells with empty cells acting as “stop”) the same demonstrations give 8.0, 11.3, 18.7, 19.7 and 22.5 successes: there, 80 demonstrations are no better than 20, so the plateau is a property of the learner, not only of the data. The learning curve says how a given learner uses data; it cannot say what the data lacks.

2. Diversity of starts

Forty demonstrations each from three start regions (means of 6 sets): from 'easy' starts (far from the pillar line) only, success is 63.7 on all and 21.8 on near starts; from 'near' only, 59.3 on all and 99.8 on near; from 'all', 98.3 and 97.8. The same number of demonstrations is worth very different amounts depending on where they start. A set must include the conditions in use, near-pillar starts included.

3. Consistency

The action variance in nearby states (pairs from different episodes within 0.25 m, 40 demonstrations): the expert from 'all' starts gives 0.0098 over 17,877 pairs, and from 'near' starts 0.0101. When 25% of the episodes are made by a careless operator who heads straight for the goal and ignores the pillar, it is 0.0304. With two operators who go round the pillar on opposite sides (alternating episodes from 'near' starts) it is 0.1437, about 14 times the consistent value. With 30% of steps replaced by a pause (zero action), it is 0.4029. These are not equally harmful, as the next parts show.

4. Mistakes and failed demonstrations

If a quarter of the episodes come from the careless operator, 7.3 of 40 demonstrations crash on average (6 sets), and a clone trained on all of them reaches 81.3 on all starts and 58.7 on near starts. Discarding the 7.3 failed episodes (keeping 32.7 successes, 980 pairs) gives 98.8 and 97.2. If half the episodes are careless, 13.7 of 40 crash; unfiltered, 68.8 and 32.8; filtered, 96.3 and 88.8. Filtering on success removed most of the damage. It did not remove all: careless episodes that happened to succeed stay in the data, and the filtered set is smaller.

5. Noise, speed and pauses

Not every imperfection hurts. Adding independent Gaussian noise to the action of every step (standard deviation 0.2 or 0.4 of the full range) gives 99.8 and 100.0 success on all starts (10 sets of 40 demonstrations): the neighbours average it away, and a noisy operator even visits more varied states. Even with standard deviation 0.8, success is 99.4 on all starts. When half of the episodes are executed at half speed, the clone still succeeds from 100.0 on all starts and 99.7 on near starts; at half speed throughout, 100 on both, but it needs 2,387 pairs for 40 demonstrations instead of about 1,200. The robot is slow, not wrong. What does hurt is replacing 30% of the steps by pauses: the clone falls to 86.0 on all starts and 87.3 on near starts (6 sets), because the pause enters the data as an action to repeat in states where the expert would move.

What the simulation simplifies. The noise is independent between steps; real operator error is correlated over time (a slow drift or an overshoot) and does not average away. There is no latency between seeing and acting, no sensor noise and no fatigue. The demonstrator is a function of the observation, so its decisions never depend on information the logs do not contain. The clone is a nearest-neighbour regressor, not a neural network, so the sample counts in the curve do not carry over to image-based policies, which typically need many more demonstrations. The experiment shows which problems can matter and how to measure them, not how much they matter on a real robot.

Engineering implications and limitations

Before recording

• Write a protocol. Where to start, how fast to move, what counts as a good episode, when to discard. Two operators following one written protocol disagree less than two following their instincts.
• Sample starts on purpose. Make a list of conditions (start positions, object placements, lighting) and cover it, including the hard ones, instead of recording whatever is convenient.
• Use the interface the robot can follow. Teleoperation by a spacemouse, a leader arm, a VR controller or kinesthetic teaching each give different action spaces and delays. Record the action the robot is commanded, not an approximation, and match the control rate you will use at deployment.
• Keep operators few and trained. A new operator is a new source of inconsistency; if you need several, measure the variance between them.

While recording

• Log everything needed to audit it. Episode id, operator, interface, date, robot and software version, start, seed, the full observation and action at every step with a timestamp, the outcome and any operator note ("slipped", "object dropped").
• Mark, do not delete. Record failed episodes with a flag. They are useful as negative examples later and as evidence of the demonstrator’s success rate.
• Watch the first episodes with the policy’s eyes. Replay a few as the policy would see them: are the observations what you intended, are actions aligned in time with what was seen?

After recording

1. Compute the three metrics: coverage, action variance in nearby states and success rate of the demonstrations.
2. Remove or flag failed episodes, and trim the idle start and end where the operator is not acting. Do not cut pauses in the middle without thinking: a pause may be the operator waiting for a reason the observations contain.
3. Look at the worst nearby-state variances: they point to disagreements to resolve (lesson M3-T6, lab M3-L4 clean-the-demonstrations).
4. Split by episode, not by step, and by operator or day if you want to know how the clone treats a new one (lesson M2-T6).
5. Plot the learning curve at 25%, 50% and 100%. If it is flat, add different data, not more of the same.
6. After training, run DAgger-style corrections where the policy is worst (lesson M3-T4, lab M3-L3 add-dagger-corrections).

Common misconceptions

Engineering takeaways

1. A good set has coverage, consistency, quality, diversity and enough quantity, and is logged so it can be audited.
2. State coverage, action variance in nearby states and the demonstrations’ success rate can be computed from the logs before any training.
3. On Scout, a nearest-neighbour clone went from 70.8 to 100.0 successes of 100 between 5 and 80 demonstrations, and the plateau began near 20 to 40; a coarser learner plateaued at 22.5.
4. Disagreement between operators (variance 0.14 against 0.01) and pauses hurt more than independent noise, and filtering failed demonstrations lifted a damaged clone from 81.3 to 98.8.
5. Start from the conditions the robot will meet: 40 demonstrations from easy starts gave 21.8 on near starts, 40 from all starts 97.8.

References and further reading

The analysis of imitation error, a noise-injection method for demonstrators, and an empirical study of learning from human demonstrations for manipulation.

• S. Ross, G. Gordon and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, AISTATS, 2011.
• M. Laskey, J. Lee, R. Fox, A. Dragan and K. Goldberg, “DART: Noise Injection for Robust Imitation Learning”, CoRL, 2017.
• A. Mandlekar et al., “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation”, CoRL, 2021.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: when two operators disagree, the average of their actions can be a crash. The last lesson of the module shows how and what to use instead of mean squared error.

Related content

Nothing below is required to finish this lesson.

• Before
  M3-T4: DAgger and Interactive Imitation

  Describe how querying the expert on the learner's own states removes the drift, and what it costs.

• Lab
  M3-L4: Clean the Demonstrations

  Two operators disagree around the pillar and the clone freezes: find the conflict and rebuild the dataset.