M1-T6 — Why Learned Policies Fail

Learning objectives

After this lesson, you should be able to:

• Name seven failure modes of learned policies and a visible symptom of each.
• Choose the first diagnostic check for a symptom.
• Diagnose three linear policies for Scout from their success rates on different sets of starts.
• Say why a failure is often two causes at once and how to separate them.

The intuition

Think of a doctor seeing a patient with a cough. The cough is a symptom; there are a few common causes; each has a quick test. Nobody orders every test, and nobody treats a cough with the first pill in the cupboard. The same method works for policies. “Success is 45%” is the cough. The question is which of the usual causes it is.

A useful habit is to separate three layers. Is the data wrong or missing (what the policy learned from)? Is the learning wrong (the optimisation, the model, the loss)? Is the evaluation wrong (how you decided it failed)? The first check for most failures is to find the layer, and the cheapest way is to watch a few rollouts and look at the numbers on the same starts the policy trained on.

The mathematics

A field guide

The table gives a symptom you can see, the first check, and where the track treats the cause in depth. None of these is a certain diagnosis; they are where to look first.

Two quantities to compute first

Whatever the symptom, two numbers narrow the search: the loss or error on the training data, and the success on held-out starts.

Why the learner’s own errors matter

A policy that is wrong by a small amount at each step drifts. If the horizon is T steps and the per-step error is ε, then behaviour cloning can accumulate an error that grows like ε·T², not ε·T, because each mistake moves the robot to states the expert never showed (Ross, Gordon and Bagnell, 2011). The details come in lesson M3-T3; the point here is that low training error does not bound the error in deployment.

A worked example

All numbers come from running RLCore on Scout’s world. Three linear policies are fitted by ridge regression to the same 945 steps from 30 expert demonstrations (starts from starts(1, 30, 'all')). Each is scored on 100 held-out starts per set from seed 500. The only difference is the feature set.

Raw features: it cannot represent the behaviour. The expert’s action is a unit-length direction, which does not change with distance, but a linear map of [dx, dy] changes with distance. The fit ends up almost constant: about 0.86 in x and a weak pull in y. Started at (1, 1) the policy drives straight east into the wall at x = 10, 1.15 m short of the goal, and stays there for all 80 steps, return 80 × (−0.05) = −4.00. From easy starts it never crashes and never succeeds, return −4.00 every time; from starts near the pillar line it hits the pillar in 100 of 100. Training error is high and so are the failures: underfitting. The fix is in the features, not in training longer.

Unit direction: it fits the goal, not the pillar. Training error drops from 0.0745 to 0.0593 and the policy now heads for the goal from every start. From easy starts it succeeds 88 times in 100, with 2 crashes. But the expert also swerves round the pillar, and no linear function of the goal direction can do that. From starts near the pillar line it crashes in 100 of 100. On all starts it succeeds in 45 and crashes in 52. This looks like distribution shift but the root cause is representation: the information needed (where the pillar is) is not in the features. Collecting more demonstrations would not help; the first check, plotting where the failures start, shows them all on one line through the pillar.

Pillar-aware features: it works, so check the evaluation too. Training MSE 0.0088, 30 of 30 on training starts, 100 of 100 on every held-out set (Wilson [0.963, 1.000] each). Mean return is 8.45 against the expert’s 8.46 on the same 100 starts, so it is as good as the demonstrator. Before celebrating, ask what the world does not test: one obstacle, no noise, no delay, no heading. In this hall the answer is honest but narrow.

The lesson of the table. Three policies, one learner, one dataset, three different failures. Training success alone (0, 10, 30 out of 30) already ranks them correctly here, but only the held-out column against the near-pillar set shows why the middle one fails. Scores on an average of starts (45 of 100) conceal a policy that is sound in one region and hopeless in another.

What the simulation simplifies. RLCore is deterministic and has no sensing noise, so failures of instability, reward hacking and the simulation gap cannot appear in it; they are the subject of the labs of later modules. Lab M1-L4 shows them in recorded runs instead.

Engineering implications and limitations

Watch before you plot

Ten minutes of watching rollouts, especially the worst and the highest-return ones, finds more than an hour of curves. Log enough to replay an episode: seed, start, observations, actions, rewards.

Change one thing at a time

A fix that changes the features, the step size and the dataset at once cannot be credited to any of them. Make a table of runs and keep it.

Run a baseline that cannot fail

The expert, or a hand-written controller, run through the same evaluation code, shows whether the evaluation itself is sound. In Scout’s hall the expert reaches the goal from every start, so a failure in the same harness belongs to the policy.

Several causes at once

Failures stack. A policy can have a poor reward, an unstable optimiser and a leaky split together, and fixing the loudest one reveals the next. Treat diagnosis as repeating the loop, not as a single answer.

Keep a failure log

Record each failure with its symptom, the check that identified it and the fix. In a few months it is the best document you have.

Common misconceptions

Engineering takeaways

1. Failures cluster into distribution shift, reward hacking, overfitting, instability, bad data, simulation gap and evaluation error.
2. Each has a visible symptom and a cheap first check; look at rollouts and compare training with held-out scores first.
3. Three linear policies on one dataset gave 0%, 45% and 100% because of the features, with different causes.
4. Averaged scores hide regional failure: report the stratum where the policy is worst.
5. The track revisits each mode: M2-T5, M3-T3, M5-T4, M6, M8, M10 and M11.

References and further reading

The imitation-learning analysis behind compounding errors, a source on gaming specifications, and the evaluation papers.

• S. Ross, G. Gordon and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, AISTATS, 2011.
• V. Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020.
• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning That Matters”, AAAI, 2018.
• R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction”, 2nd ed., MIT Press, 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the labs of this module put these ideas on Scout: record a rollout, learn to reach the goal, score a policy fairly, and read a failed training run.

Related content

Nothing below is required to finish this lesson.

• Before
  M1-T5: Evaluating a Policy

  Measure success rate and return on held-out scenarios, and report them with seeds and confidence intervals.

• Lab
  M1-L4: Read a Failed Training Run

  Four runs went wrong in four ways: read the curves and the rollouts and name the failure mode in each.