M11-T1 — Baselines and Ablations

Learning objectives

After this lesson, you should be able to:

• List the baselines a result should be compared against (trivial, hand-coded, simple learned, reference) and state what each one rules out.
• Design an ablation with leave-one-out and add-one variants, and explain why the two can disagree when components are redundant.
• Give a baseline the same data, tuning budget, evaluation starts and number of seeds as the method, and say what the comparison costs each side.
• Write each comparison as a claim with its evidence, so a reader can see what was and was not tested.

The intuition

Take Scout's reach task from Module 1: a 10 m by 6 m hall, a goal at (9, 3), a pillar at (5, 3), and 100 start positions drawn with RLCore.starts(900, 100, 'all'). The figure shows nine ways to produce a policy, each scored on those same 100 starts. Doing nothing and acting at random score 0 of 100. A hand-coded rule that heads straight for the goal and ignores the pillar scores 28 (95% interval 0.20 to 0.37): it mostly gets through from starts away from the line through the pillar (87 of 100 easy starts) and never from near it. Linear behaviour cloning on the raw direction to the goal scores 0, on a normalised direction 40 of 100, and on features that include the pillar 100 of 100, the same as the expert it copies. The surprise is the nearest-neighbour row: a 1-nearest-neighbour policy that looks up the closest of the 945 demonstrated observations and copies its action, with no model and no training, scores 99 of 100.

The bottom row is PPO from lesson M6-T5 (60 iterations of 12 episodes) over 20 training seeds. Its dots are not near one number: ten seeds end at 0, the other ten between 0.52 and 1.00, and the mean is 0.47. "It scores 0.47" is a poor summary and "its best seed scores 1.00" a worse one (lesson M11-T2). Set against the baselines, the honest reading is: given an expert to copy, a nearest-neighbour lookup is a stronger and far cheaper entry on this toy task than 720 training episodes of PPO. That is no verdict on PPO, but it is a fact a paper should have to face.

The mathematics

A comparison is more than two numbers

Write a result as a score s(M, D, B, E): a method M, given data D (demonstrations, simulator, reward), a tuning budget B, and evaluated under a protocol E (the start positions, the number of episodes and seeds, the policy mode). A comparison of M₁ against M₂ is fair only if D, B and E are the same, or if each difference is stated and priced in. Of the three, B is the one most often unequal: the new method is tuned, the baseline is left at its default. The score of a policy over n evaluation episodes with k successes is the success rate p = k/n, reported with its 95% Wilson interval (lesson M1-T5); when the unit is a training run, seeds replace episodes (lesson M11-T2).

Which baselines

Ablations

Let a method consist of components c₁, …, c_m (for PPO: the advantage estimator, the clipped objective, several epochs per batch; for domain randomisation: which factors are randomised). Let s(S) be the score with the set S of components switched on, so s(all) is the full method and s(none) a bare baseline. Two effects of component i are:

If components are independent and additive, d_i^- and d_i⁺ agree. When two components do the same job (redundancy), removing either leaves the other and d_i^- is about 0 for both, while d_i⁺ is large for both. When a component only helps in the presence of another (a prerequisite), d_i⁺ is about 0 and d_i^- is large. Neither summary alone is the truth; a table with both, and ideally a few pairs, is. A third kind of study, vary one, sweeps a single setting (the GAE parameter λ, say) and plots the score against it.

M, D, B, E

Method, data, tuning budget and evaluation protocol that together define a score. Two scores are comparable when these match.

s(S)

Score (here the success rate) of the method with the set S of components switched on.

d_i⁻, d_i⁺

Leave-one-out and add-one effect of component i.

seed

The integer that fixes the random numbers of one training run (initialisation, exploration noise, training starts). The seed is the unit of replication of a learning method.

Every row of a results table should read as a sentence: on task T, with data D and budget B, M scores X (interval) against Y (interval) for the baseline. Anything stronger ("M is better", "component i is necessary") needs the statistics of lesson M11-T2 and, for "necessary", the add-one evidence too.

A worked example

All numbers come from Node scripts against RLCore (scratch prototypes). Evaluation starts are RLCore.starts(900, 100, 'all') unless stated, and policies are deterministic (PPO uses its mean action). Demonstrations are expert rollouts from RLCore.starts(1, 30, 'all'), 945 observation-action pairs; "ridge" is RLCore.ridgeFit (penalty 0.001) on the named features of Module 3; "1-nearest neighbour" copies the action of the closest demonstrated observation [dx, dy, px, py]. The random policy draws each action uniformly in [-1, 1].

Baselines on the reach task

By hand, the interval for 99 of 100: p = 0.99 and z² = 3.8416. The denominator is 1 + 3.8416/100 = 1.0384. The centre is (0.99 + 3.8416/200)/1.0384 = 1.0092/1.0384 = 0.9719. The half-width is 1.96 · √(0.99 · 0.01/100 + 3.8416/40000)/1.0384 = 1.96 · √0.0001950/1.0384 = 0.0264. The interval is 0.946 to 0.998, matching the program. For the perfect score of 100 of 100 it is 0.963 to 1.000, which overlaps it: with 100 evaluation episodes the nearest-neighbour baseline and the expert are not distinguishable.

Reading the table. The trivial baselines confirm the task is not solved by chance. The straight-line rule (28 of 100, 72 crashes) shows the pillar is the whole difficulty: 0 of 100 near the pillar, 87 of 100 on easy starts, so a method evaluated only on easy starts would hardly be tested. Linear cloning of a normalised direction (40) cannot represent "swing round the pillar", and cloning on raw [dx, dy] (0) is worse than the rule: the fitted map is almost a constant push to the right (bias 0.86, slopes below 0.1), so on easy starts Scout ends its 80 steps 1.0 to 1.2 m from the goal (checked on five starts) and times out. Adding the pillar features gives 100 of 100, and the nearest-neighbour lookup needs no features at all and scores 99.

How stable are these baselines? I repeated the demonstration collection with ten start seeds (RLCore.starts(s, 30, 'all'), s = 1 to 10) and re-scored on the same 100 starts. Ridge on pillar features: 100 of 100 every time. 1-nearest neighbour: 99, 100, 100, 100, 97, 100, 99, 100, 99, 100 (mean 99.3). Ridge on direction features: 40, 34, 45, 43, 42, 37, 46, 42, 43, 44 (mean 41.6). With only 5 demonstrations ridge on pillar features scores 99 of 100, with 10 it scores 100. The strong baselines are strong across demonstration sets; their cost is one expert and 5 to 30 demonstrated episodes.

PPO costs something different: no expert, but 720 training episodes (60 iterations of 12). Over seeds 1 to 20 (RLCore.trainPolicyGradient({algo: 'ppo', iters: 60, eps: 12, seed: s}, rng(s))) the seed-level success was 0 on ten seeds and 0.52, 0.82, 0.97, 0.99 and six times 1.00 on the others. The mean, 0.465, is below every strong baseline and the best seed matches them; neither is "the performance of PPO" (lesson M11-T2). If no expert exists the cloned baselines are unavailable and the comparison changes. State which.

Ablating PPO: which part matters?

Six variants, each run for seeds 1 to 20 (60 iterations of 12 episodes) and scored on the same 100 starts. "No clip" sets the clip range to 100; "A2C" takes one gradient step per batch with no clipping (lesson M6-T4); "REINFORCE" uses returns-to-go and no critic (lesson M6-T2); λ = 0 and λ = 1 change the GAE parameter of lesson M6-T3 from its default 0.95. The p-value is the two-sided permutation test of lesson M11-T2 against the default (10000 shuffles).

What the table supports. Only the advantage estimate stands out: the one-step advantage (λ = 0) loses most of the success, 0.08 against 0.47, an effect of about one pooled standard deviation. The other differences are inside the seed noise. Removing the clip raised the mean to 0.63 (p = 0.29): it would be wrong to write "clipping hurts", and equally wrong to write "clipping is unnecessary", because a 60-iteration run may not exercise what the clip is for (stability over long runs, lessons M6-T5 and M6-T7). With three seeds the picture would have been clean and false: the first three seeds scored 0.97, 0.82, 0.00 (mean 0.60) for the default and 0, 0, 0 for λ = 0, but 0.95, 0.00, 1.00 (mean 0.65) for no clip, which would have made the clip look useful. The t-interval of λ = 0 dips below 0 because it does not know a rate cannot be negative (bootstrap: 0.00 to 0.21); lesson M11-T2 returns to this.

Ablating domain randomisation: leave-one-out against add-one

The Module 10 toy has a three-number policy theta (RLCore.trainTheta, a 15-candidate grid search) and a floor world with gain 2.2, drag 3.4, delay 1, observation noise 0.05 and process noise 0.03 (lesson M10-T2). Full randomisation draws gain in [2, 3.2], drag in [2.8, 3.8], delay in {0, 1, 2}, observation noise in [0, 0.08] and process noise in [0, 0.04] each training episode. For each variant I trained ten thetas (seeds 1 to 10) and scored each on the floor over 5 seeds of 40 episodes (200 episodes).

Read leave-one-out alone and no factor matters: the full recipe minus any one factor stays at 0.995 or higher. Read add-one alone and the delay does all the work (1.000, against 0.725 from process noise only, the same as no randomisation), with gain and drag nearly as good. Together they say delay and gain-and-drag are redundant here: either prepares the grid search for this floor. Leave-one-out alone would have said the randomisation is unnecessary; the full recipe alone would have credited every factor. The study says which factors suffice for this simulated floor, not for a real one.

Engineering implications and limitations

Baseline and ablation hygiene

• Include the cheapest thing a sceptic would try: a hand-coded rule, a nearest-neighbour lookup, a linear model. If the method does not clearly beat them, the paper is the baseline.
• Same data, same evaluation, same tuning effort. Same demonstrations or environment steps, start positions, episodes, seeds and policy mode; choose the number of hyperparameter settings per method in advance and state it. Untuned baselines are a common route to an inflated gain, and implementation details matter more than they should (Henderson et al.; Engstrom et al.).
• Price the comparison. The cloned baselines need an expert and 30 demonstrations, PPO a reward, a simulator and 720 episodes. Report data, episodes and compute next to the score (lesson M8-T5), and keep the embarrassing baseline in the report.
• Change one thing at a time and say what is fixed: "A2C" changed epochs and clipping together, so its row cannot say which matters. Run leave-one-out and add-one; retune settings that interact with the component.
• Use enough seeds for the spread (lesson M11-T2), and match the claim to the evidence: "removing X changed success from a to b over n seeds" is a result, "X is essential" also needs the add-one evidence.

A table to put in every report

What this toy cannot tell you

Scout is a point in a hall with one pillar. The nearest-neighbour baseline works because the demonstrations cover the same region of observation space the policy is later asked about; with a richer observation (images, lesson M9-T1) or a start distribution the demonstrations did not cover (lessons M3-T3 and M8-T2) it would fail, and a comparison on a larger task could rank the methods differently. The PPO budget is small on purpose, so the ablation says something about short training runs and not about long ones. No real robot data is used anywhere in this lesson.

Common misconceptions

Engineering takeaways

1. A baseline is the cheapest alternative a sceptic would try; on the Scout reach task that includes a hand-coded rule (28 of 100), ridge regression on good features (100 of 100) and 1-nearest-neighbour cloning (99 of 100).
2. Give baselines and the method the same data, the same evaluation starts, the same number of seeds and the same tuning effort, and state the cost of each: 30 demonstrations and an expert, or 720 training episodes and a reward.
3. Ablate in both directions: leave-one-out shows what the method cannot lose, add-one shows what suffices. In the toy randomisation study they disagreed (all but delay 0.995, delay only 1.000, none 0.725) because the factors were redundant.
4. In 20 seeds of short PPO runs only the advantage estimate stood out (λ = 0 scored 0.08 against 0.47); clipping, λ = 1, A2C and REINFORCE differences were within the seed-to-seed noise, and three seeds would have shown clear but false effects.
5. Write each comparison as a claim with its evidence and its limits; all studies here are toy studies on the simulated Scout, and no real robot data was used.

References and further reading

The k-nearest-neighbour and ridge baselines are standard textbook methods and the PPO variants follow the descriptions of lessons M6-T2 to M6-T5; the numbers are from my own scratch experiments on the toy Scout world, not from the papers below. The papers discuss, for deep reinforcement learning on standard benchmarks, how baselines, implementation details and seeds affect reported gains.

• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning that Matters”, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
• L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph and A. Madry, “Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO”, International Conference on Learning Representations (ICLR), 2020.
• A. Rajeswaran, K. Lowrey, E. Todorov and S. Kakade, “Towards Generalization and Simplicity in Continuous Control”, Advances in Neural Information Processing Systems (NeurIPS), 2017.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Lesson M11-T2 takes the seed-to-seed spread seen here seriously: why learners are bimodal, which interval to use for ten or twenty seeds, how a permutation test and an effect size read, and why reporting the best seed misleads.

Related content

Nothing below is required to finish this lesson.

• Lab
  M11-L1: Compare Two Policies Statistically

  Two policies, ten seeds each: decide whether one is better, with intervals and a permutation test.