Robot Learning · Theory · M6
M6-T7 — Training Stability and Diagnosis
A training run is an experiment with instruments. Learn what each instrument reads when something is wrong.
A PPO run that fails usually fails silently: the return sits where it started and nothing crashes. The cause might be a learning rate, a critic that learns nothing, too little data per batch, or just a bad seed. Each leaves a different trace in the numbers the trainer logs every iteration. This lesson names those numbers, shows what each failure does to them in real runs of this module’s trainer, measures how far two identical runs can differ, and gives an order in which to look. Labs M6-L3 and M6-L4 put it into practice.
Learning objectives
After this lesson, you should be able to:
- Name the eight diagnostics the trainer logs and say what each is evidence for and what it cannot show.
- Recognise the signatures of too high or too low a learning rate, a collapsed or a tiny action std, a critic that does not learn, and too few episodes per batch.
- Report seed variance honestly: run several seeds, give the spread, and decide when one failure is evidence.
- Follow a debugging order (return, advantages, ratio and KL, entropy) and record a run so it can be repeated.
The intuition
A doctor reads a patient from several vitals at once, because one number rarely names the illness. A training run is the same. The return says that something is wrong; the other columns say where. A flat return with a huge KL and a clip fraction near the maximum says the updates are too violent. A flat return with a KL of 0.00 says they are too timid. A flat return with a perfectly healthy KL and an explained variance of −10 says the critic is not learning and the advantages are noise.
The second lesson of the figure is less comfortable: the same settings can succeed or fail depending on the seed. Ten runs of the default PPO below split into seven that reach 0.80 or more and three that end near zero. Without that spread in mind, one lucky or unlucky run reads as a finding.
The mathematics
The eight diagnostics
The trainer RLCore.trainPolicyGradient logs one record per iteration. Here is what each means, and the one thing each cannot tell you.
| Diagnostic | What it is | Reads as | Cannot tell you |
|---|---|---|---|
| return (ret) | mean undiscounted episode return in the batch | overall progress. −4.00 is exactly 80 steps at −0.05, an episode that times out. About +8 is a fast goal. | why; also noisy with 12 episodes |
| success | fraction of the batch’s episodes that reached the goal | the task metric, on the exploring (stochastic) policy | how the mean policy would do; evaluate that separately |
| entropy | entropy of the Gaussian policy in nats, summed over the 2 action dimensions | how random the policy is. 1.45 at std 0.5; −3.15 at std 0.05. | whether the randomness is useful |
| KL | mean of log πold − log πnew on the batch after the update: how far the policy moved | 0.01 to 0.05 is a gentle step here. Hundreds is a leap. | whether the move was in a good direction |
| clip fraction | share of samples whose ratio was outside the clip range and therefore got no gradient | how many samples the clip silenced: near 0 means timid or lr tiny, large means the policy moved a lot within the epochs | which samples (lab M6-L2 looks at this) |
| value loss | mean squared error of the critic on its targets | critic training. It can be large for a good reason: when the policy suddenly finds the goal, returns jump. | whether the critic is useful: it can be small because everything is constant |
| explained variance (ev) | 1 − Var(returns − values) / Var(returns) | 1 is a perfect critic; 0 is no better than the mean; negative is worse than a constant | anything when the returns are constant (the trainer logs 0 below a tiny variance) |
| action std | mean standard deviation of the Gaussian over the 2 dimensions | the same information as entropy, in units of the action | the same limits |
Two definitions are used all through the lesson. The KL printed is the simple sample estimate of the divergence of the old policy from the new one, which can be slightly negative from noise (lesson M6-T5). Explained variance is the quantity you implement in lab M6-L1.
Failure patterns and their signatures
A signature is a combination of columns, not a single column. The table summarises the runs described in the example; the numbers are from this module’s trainer, with seed 1 unless stated.
| Failure | Return | KL / clip fraction | Entropy / std | Critic (ev, value loss) | First thing to try |
|---|---|---|---|---|---|
| learning rate too high | stuck at −4 or oscillating | KL in the hundreds, clip fraction 0.4 to 0.7 | drifts up, not down | ev very negative | lr down by 10× |
| learning rate too low | flat at −4 for the whole budget | KL 0.00, clip fraction 0.00 | does not move | slowly improving | lr up, or more iterations |
| tiny action std (entropy collapsed) | flat at −4.0, never finds the goal | normal (0.02, 0.1) | entropy −3.15, std 0.05 from iteration 0 | looks fine: ev is high because returns are constant | raise std or the entropy bonus |
| critic that does not learn | flat, or lucky | normal | normal | ev −10, value loss flat near 0.44 | critic lr or weight; check the loss is applied |
| too few episodes per batch | jumps between −4 and −5 (crashes) | noisy, clip fraction somewhat higher (0.13 against 0.09) | normal | ev noisy | more episodes per batch |
| unlucky seed | flat at −4.0 | normal | normal | normal | run more seeds before changing anything |
Two failures in the usual lists could not be shown with this trainer, and I do not invent numbers for them. Reward scale: the critic’s targets and its loss scale with the reward (a reward ten times larger gives a value loss a hundred times larger), and an optimiser sees badly scaled targets as a learning-rate problem. Observation scaling: the trainer divides the observation by (5, 3, 5, 3) so each input is about ±1; a network fed raw metres or millimetres learns slowly or saturates its tanh units (lesson M2-T2, features and normalization). These are the first checks when a critic’s loss is large and not falling.
Seed variance
The seed selects the start positions of each iteration and the network initialisation, and with it the noise of every rollout. An RL run is therefore a random variable, and one run is one sample from it. Report the same quantities as for any experiment: the number of seeds, the mean, the spread and the individual values (lesson M11-T2). With a binary outcome such as "reached the goal", a Wilson interval (lesson M1-T5) gives a fair bound even for ten seeds.
A worked example
Every number below is a real output of RLCore.trainPolicyGradient in Node, with a seeded generator (SLAMCore.rng(seed)), the default 60 iterations of 12 episodes unless stated. "Final" values are means of the last 10 iterations. "Held-out success" is the deterministic mean policy (RLCore.meanPolicy) on 50 new starts from starts(777, 50, 'all'). Only the named setting differs from the default.
1. The baseline, seed 1. The return sits at −4.000 and success at 0 until about iteration 40, then climbs: at iteration 49 the return is 7.171 and success 0.917. At the iterations I printed (every fifth to tenth), KL is 0.003 to 0.06, the clip fraction 0.01 to 0.28, entropy 1.452 and std 0.500. The value loss rises when the goal is found, from 0.05 at iteration 29 to 6.03 at iteration 49, and ev falls to 0.03: the critic is suddenly asked to predict returns of +9 where it only ever saw −4, a large error for a good reason. Final: return 6.04, success 0.82, ev 0.13, held-out success 0.92. A rising value loss is a reason to look, not a sign of failure.
2. Learning rate too high (lr 0.1, 33 times the default). In the first iteration the KL is 211.4 and the clip fraction 0.647; ev is −25.4. By iteration 14, KL is 293. Final: return −4.22, KL 1.34, clip fraction 0.42, ev −1.18, held-out success 0.00. Over seeds 1 to 5 all five have held-out success 0.00, mean final clip fraction 0.26, mean ev −11.2. The entropy rises (1.63, std 0.546) rather than falling. Note what is not a sign: lr 0.05 also had a first-iteration KL of 75.0 and a clip fraction of 0.647, yet seed 1 reached 0.98 held-out success, and across seeds 1 to 5 it gave 3 of 5 successes (the default gave 7 of 10). A large first-iteration KL alone is not a verdict; a KL that stays in the tens to hundreds is.
3. Learning rate too low (lr 3e-4). Final KL 0.003 and clip fraction 0.005, with a clip fraction that is often exactly 0.000: the policy is barely moving. Return −3.93. Held-out success 0.00 for seed 1; 1 of 5 seeds succeeded. Compare the same two columns in the high case (KL 211, clip 0.65) and you see that those two columns, not the return, tell the two apart.
4. A tiny action standard deviation (initStd 0.05). Entropy is −3.153 and std 0.050 from the first iteration to the last, and the return is −4.00 (seed 1) with zero success on every iteration. All five seeds: held-out success 0.00. Everything else looks healthy: KL 0.02, clip fraction 0.1, ev 0.73 on average, because a critic that predicts "−4 forever" is nearly right. With a std of 0.05 the policy’s action noise moves Scout 1.25 cm per step, so it never wanders into the goal metres away, and without a goal reward the advantages carry nothing about it. Honest caveat: this is a collapse at the start. In 60 iterations the trainer’s entropy never fell during training in any run I made (1.452 to 1.451 for the default; 1.43 even at lr 0.05), because its std is a learned parameter with a deliberately slow update. A run where entropy falls by itself and strangles exploration, as PPO runs sometimes do with long training or small batches, was not reproduced here.
5. A critic that does not learn (vcoef 0). Setting the critic weight to 0 removes the critic’s gradient entirely, so it stays at its random initial values. Value loss is flat at 0.44 (seed 1: 0.428 at iteration 0, 0.439 at iteration 59), ev is −10.2 at the end (−14 at the start), and the return is −4.00; held-out success 0.00. Over seeds 1 to 5, mean ev −7.3 and one seed in five succeeded (that seed had value loss 19.7 and ev −1.33). Advantages from a bad critic are mostly noise (lesson M6-T3); the one success shows that a lucky stretch can still get there. Value loss alone (0.44) looks small: it was ev, not the loss, that shouted.
6. Too few episodes per batch (eps 4). With 4 episodes the logged return of seed 1 wanders: −4.65, −5.05, −4.00, −5.01, −4.33 and −4.55 at iterations 14, 19, 29, 39, 49 and 59. Returns below −4 mean crashes (a crash costs at least −5.05). Held-out success 0.00 for seed 1, and 1 of 5 seeds succeeded, against 7 of 10 with 12 episodes. With so few seeds the gap could be chance; the Wilson intervals (0.04 to 0.62 against 0.40 to 0.89) overlap. The cleaner evidence is mechanical: 4 episodes of 80 steps give at most 320 samples to estimate an advantage and a gradient, and in a task where goal-reaching is rare most batches contain no goal at all.
7. Ten seeds of the default PPO. Held-out success: 0.92, 0.80, 0.00, 1.00, 1.00, 0.04, 1.00, 0.00, 1.00, 1.00. Mean 0.68, standard deviation 0.46 (the 95% interval of the mean from a t table extends above 1, which is impossible for a success rate, a sign that the average of such a bimodal outcome is a poor summary). 7 of 10 seeds reach at least 0.80 (Wilson 95%: 0.40 to 0.89). The seven successful runs first exceeded 0.8 training success at iterations 5, 14, 30, 32, 36, 42 and 46: the time to success varied nine-fold. The three failures (seeds 3, 6, 8) have final returns of −4.06, −4.13 and −4.02, almost entirely timeouts. I reran them for 150 iterations: still success 0.00 and held-out success 0.00, so these are not just slow. I did not diagnose why; my guess is that early batches contained no goal-reaching episodes and the policy drifted somewhere unhelpful, which I have not checked.
What this simulation simplifies. One small task, one network size, only a handful of settings and 5 to 10 seeds per setting. The effect sizes would change with other tasks. The goal is to show the instruments, not to rank the settings.
Engineering implications and limitations
The order in which to look
- The return and success first. Is it flat at the timeout value (−4.00), noisy, rising then collapsing, or fine on training and poor on held-out starts? Evaluate the mean policy on held-out starts too (lesson M1-T5; failure modes in lesson M1-T6): training success is the exploring policy.
- Then the advantages. Is the critic any good (ev above 0)? Are advantages normalised? Is a time-limit truncation treated as a terminal (lab M6-L1)? If the advantages are noise, nothing after them can work.
- Then the ratio, KL and clip fraction. Is the policy moving by tens of nats or by 0.00? Compare with the table of signatures.
- Then the entropy and std. Is the policy exploring enough to ever find the goal, and is that changing?
- Then the seed. Run at least five seeds before you call a setting good or bad.
Change one thing at a time, and write down the change. Lab M6-L3 (tune a PPO run) asks you to do this with the learning rate, clip range, epochs, batch size and entropy bonus; lab M6-L4 (diagnose a diverging training run) gives you four runs with their logs and asks for the cause of each.
Logging practice
- Log every diagnostic at every iteration, not only the return, plus the evaluation of the mean policy on a fixed set of held-out starts every few iterations.
- Plot them on a shared x axis, with the return and success on top and KL, clip fraction and ev beneath, so a change in one lines up with a change in another.
- Record the time taken: an iteration that takes ten times as long is also a symptom.
- Keep the logs of failed runs. They are the data for the next diagnosis, and for the honest reporting of how many runs it took.
Reproducibility
A run is reproducible when someone else can get the same numbers. Record the seed, every option of the trainer (algo, iters, eps, epochs, gamma, lam, clip, lr, vcoef, entCoef, hidden, initStd, region) and the version of the code. In this trainer all randomness comes from the generator passed in and the seed selects the starts, so the same options and seed give the same hist exactly. On real hardware (GPU kernels, parallel workers) bit-for-bit repeatability is often not achievable; there you report the spread over seeds instead (lesson M11-T3).
What to do about a bad seed
Do not discard the failures and report the best run: that is selecting on the outcome. Report the number of seeds and how many succeeded, and if a failure rate of 3 in 10 is unacceptable, treat it as a problem to fix: a larger batch, more iterations, an entropy bonus, easier starts such as the 'easy' and 'near' regions of starts(), or a different reward.
Common misconceptions
“The return is flat, so the learning rate must be wrong.”
A flat return at −4.00 appeared with a learning rate 33 times too high, 10 times too low, a tiny std, a dead critic, too few episodes, and a bad seed. Only the other columns tell them apart.
“A huge KL always means the run will fail.”
At lr 0.05 the first iteration had KL 75 and the run still ended at 0.98 held-out success. It is the KL that stays large, with a high clip fraction, that goes with failure.
“A low value loss means the critic is good.”
With the critic switched off the value loss was a flat 0.44; with the returns constant at −4 the loss is small whatever the critic knows. Explained variance is the better check, and it too is uninformative when the returns do not vary.
“I ran it once and it worked, so the setting is good.”
The default setting reached 0.80 or more on seven of ten seeds and about 0.00 on three. One run is one draw.
Engineering takeaways
- Read the diagnostics as a set: return says that something is wrong, KL and clip fraction say whether updates are too big or too small, entropy says whether the policy can still explore, ev says whether the advantages mean anything.
- Several different failures give the same flat return of −4.00 here; their other columns differ (KL 211 against 0.00, entropy −3.15, ev −10, return jumping to −5).
- Across 10 seeds the default PPO had held-out success 0.92, 0.80, 0.00, 1.00, 1.00, 0.04, 1.00, 0.00, 1.00, 1.00 (mean 0.68, sd 0.46): report spread, not one run.
- Debug in order: return and success, then advantages and critic, then ratio and KL, then entropy, then the seed.
- Record the seed, the full option list and the version of the code, and keep the logs of failed runs.
References and further reading
Schulman et al. introduced PPO and the diagnostics around it; Henderson et al. is the standard warning about seed variance and reporting.
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.
- P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning that Matters”, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A run shows return −4.00, KL 0.00 and clip fraction 0.00 for the whole budget, with ev improving slowly. What is the likely cause, and what is the first change?
Explanation
The updates are too small: the policy hardly moves. Raise the learning rate (or allow more iterations or epochs). KL and clip fraction near zero distinguish it from a violent learning rate, where they are very large.
Question 2
A run has entropy −3.15 from the first iteration, return −4.00, and a healthy KL and explained variance. Why is the explained variance misleadingly good, and what do you change?
Explanation
The returns are almost all the same (a timeout), so a critic that predicts that constant is nearly right. The cause is the action std of 0.05, which never explores to the goal; raise the initial std or add an entropy bonus.
Question 3
Four seeds of a new setting succeed and one fails. Can you conclude the setting is better than one that succeeded on 7 of 10 seeds?
Explanation
No. 4 of 5 and 7 of 10 are compatible, and the Wilson intervals overlap widely (0.38 to 0.96 against 0.40 to 0.89). Run more seeds before concluding anything.
What’s next
Next: lab M6-L3 lets you tune a PPO run and read its curves, and lab M6-L4 gives you four broken runs to diagnose. Module 7 then replaces the model-free update with a learned model of the world.
Related content
Nothing below is required to finish this lesson.
- BeforeM6-T6: SAC and Off-Policy Learning
Describe replay, soft Q-functions and entropy regularization, and say when off-policy methods win.
- LabM6-L3: Tune a PPO Run
Change learning rate, clip range, batch size and entropy bonus for a simulated Scout and read the curves that follow.
- LabM6-L4: Diagnose a Diverging Training Run
The return rises and then collapses: use the logged diagnostics to find which of four causes it is.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.