M3-T1 — Learning from Demonstrations

Learning objectives

After this lesson, you should be able to:

• Say what one demonstration contains and write it as a dataset of observation-action pairs.
• Name four ways to collect demonstrations and the trade-off of each.
• Choose between imitation, reinforcement learning and a hand-written controller for a task, with reasons.
• Measure the state coverage of a set of demonstrations and say what it implies for the learned policy.

The intuition

Teaching a new colleague to push a cart around a pillar is easy by example: you walk it once, they watch, and the next time they do the same. What they copy is what you did, in the places you did it. If you only ever walked the left side of the hall, the colleague has no example of what to do on the right side. They may guess well, because they understand the purpose; a learned policy has no such understanding and only has your examples.

That is the whole idea of imitation learning and also its limit. The expert supplies a policy evaluated at the states the expert visited. A good demonstration set is dense where the expert goes and silent everywhere else, so the question to ask of any set of demonstrations is not only “was the expert good?” but “which states does this set actually show?”

The mathematics

What a demonstration is

A demonstration is a trajectory of the expert interacting with the world. At each step the learner records what the expert’s controller saw and what it commanded.

o_t

observation at step t (for Scout, [dx, dy, px, py], see lesson M1-T2)

a_t

the action the expert commanded at step t (for Scout, a velocity [ax, ay] in [-1, 1])

g

optional goal or task label, when one policy must serve many tasks

N, L

number of demonstrations and episode length; the dataset has about N · L pairs

π_E

the expert’s policy, the thing being imitated; it may be a person, a controller or a planner

Rewards are not part of a demonstration. A demonstration does not say why the expert acted; it says only what the expert did. Some datasets add the reward, the success flag or the language instruction, and later modules use them (M8 for offline reinforcement learning), but pure imitation needs only observations and actions.

Who provides demonstrations

Scout’s expert in the track is the third kind: the RLCore function expert(obs) heads for the goal and swings round the pillar on the side the robot is already on. It is deterministic, so every demonstration from the same start is identical, which real people never are.

Imitation, reinforcement learning or programming?

Relation to supervised learning

Once the pairs are recorded, imitation is a supervised-learning problem as in module M2: the observation is the input, the expert’s action is the label, and a loss compares the prediction with the label. The next lesson, M3-T2, writes this out. What makes imitation harder than ordinary supervised learning is that the policy’s own predictions change the next inputs, so the test distribution depends on the learner (lesson M3-T3).

A worked example

All numbers come from running RLCore. The demonstrations are the expert run from the 30 starts starts(1, 30, 'all'): positions in the left third of the hall, x from 0.5 to 3.5 m and y from 0.5 to 5.5 m, not inside the pillar’s collision circle.

The dataset. Each demonstration ends when the robot reaches the goal. The 30 demonstrations are all successful and contain 945 steps, 31.5 per demonstration on average, with lengths from 24 to 37 steps. The dataset is therefore 945 pairs of observation [dx, dy, px, py] and action [ax, ay]. The mean return of the demonstrations is 8.42 (lesson M1-T3 defines the return). The first demonstration starts at (1.78, 2.32); its first two steps are observation [7.22, 0.68, −3.22, −0.68] with action [1.00, 0.09], and [6.98, 0.65, −2.98, −0.65] with action [1.00, 0.09]: the expert drives east, almost straight at the goal, at the top speed.

State coverage. Divide the hall into 0.5 m cells: 20 columns by 12 rows is 240 cells, 228 of which lie outside the pillar’s collision circle. Take the observation at each of the 945 steps and ask which cell the robot stood in. The demonstrations touch 82 cells: 34.2% of the hall (36.0% of the 228 reachable cells). Split by column, 46 of those cells are in the left third (x < 3.5 m) where the starts are, 14 in the middle, and 22 in the right, where the goal is. Counting samples per cell, 21 of the 82 cells hold only one or two observations; the busiest cell holds 39.

The coverage grows slowly with more demonstrations from the same distribution of starts:

Tripling the demonstrations from 10 to 30 adds 22 cells, and tripling again adds only 12. The hall has a large region that the expert never enters because the expert never needs to: the top and bottom walls on the right, and the whole right half away from the line to the goal. Collecting more of the same kind of demonstration cannot fix that.

What it means. A clone fitted to these 945 pairs makes a prediction in every cell of the hall, but in 158 of the 240 cells it is extrapolating. If the clone ever strays into one of them, nothing in its data says what to do. Whether it recovers depends on the model class: a smooth linear model extends the expert’s trend, a nearest-neighbour model copies the closest example whether or not it is relevant. This is the root of the compounding-error problem of lesson M3-T3, and the reason lab M3-L1 (clone-the-expert) asks you to look at where the data lie before judging the clone.

What is learned. The expert in this world goes round the pillar on the side the robot starts on: 40 of its samples at x ≈ 5 are above the pillar and 38 below. A clone sees that pattern. It does not see the idea “avoid obstacles”, and nothing in the pairs tells it that the pillar is the reason. If you moved the pillar, the clone would still swerve at the old place.

What the simulation simplifies. The expert is deterministic and perfect: its demonstrations never disagree with each other and never contain mistakes. Real demonstrators are neither. A person gives a different path each time, and may pause, correct themselves or choose the other side of the pillar from the same start; lesson M3-T5 and lab M3-L4 treat these.

Engineering implications and limitations

Record what the robot will see

Log the observations the learner will have at run time: the same sensors, the same rates, the same preprocessing. If the expert saw a motion-capture position and the clone will see a camera image, the demonstration is not the dataset you need. For teleoperation, record the command the robot executed, not only the joystick position, so that the label is the action rather than an intention that was clipped or delayed.

Synchronise and keep the clock

Store timestamps from one clock, plus the control rate. A pair (o_t, a_t) only means something if the action was chosen on that observation; the next lesson shows a one-step shift.

Plan the coverage

Before collecting, list the starts, objects and conditions the robot will meet, and sample demonstrations across them. After collecting, plot the visited states as in the figure above, with dimensions that matter for the task. Gaps are warnings, not facts about the robot.

Budget the cost

A demonstration costs operator time, robot time, set-up and resets. A scripted expert is nearly free; a skilled operator on a real arm gives perhaps tens of demonstrations an hour, and several hundred to several thousand are common for visuomotor manipulation policies. The point is not a number but the shape: demonstrations cost labour for every one, whereas simulated rollouts cost compute. A cheap interface and consistent operators are often worth more than a better algorithm.

Keep the successes and the notes

Mark failed attempts and discard them, or keep them with a flag. Write down who demonstrated, when, and with which version of the interface; lesson M3-T5 returns to this.

Common misconceptions

Engineering takeaways

1. A demonstration is a trajectory of observations and actions; the dataset is the set of (o_t, a_t) pairs, and rewards are not required.
2. Demonstrations come from teleoperation, kinesthetic teaching, scripted or planned experts, or video; each has a different cost and a different mismatch with the robot.
3. Choose imitation when showing is cheaper than specifying, reinforcement learning when outcomes can be scored and tried, and programming when a controller is easy to write.
4. A clone learns the expert’s behaviour on the states in the data, not the expert’s purpose.
5. Thirty successful demonstrations of Scout (945 steps) touched 82 of the 240 hall cells (34%); coverage, not the count of demonstrations, limits what is learned.

References and further reading

A survey of learning from demonstration, an early driving example and a recent survey of imitation methods.

• B. D. Argall, S. Chernova, M. Veloso and B. Browning, “A survey of robot learning from demonstration”, Robotics and Autonomous Systems 57(5), 469–483, 2009.
• D. A. Pomerleau, “ALVINN: An Autonomous Land Vehicle in a Neural Network”, Advances in Neural Information Processing Systems 1 (NIPS 1988), 1989.
• T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel and J. Peters, “An Algorithmic Perspective on Imitation Learning”, Foundations and Trends in Robotics 7(1–2), 2018.
• A. Hussein, M. M. Gaber, E. Elyan and C. Jayne, “Imitation Learning: A Survey of Learning Methods”, ACM Computing Surveys 50(2), 2017.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lesson M3-T2, Behavior Cloning, turns these pairs into a policy by supervised learning and shows how features, model class and a one-step label error change the closed-loop result.

Related content

Nothing below is required to finish this lesson.

• Lab
  M3-L1: Clone the Expert

  Learn to follow the corridor from 40 expert demonstrations and measure how well the clone does on the same route.