M11-T2 — Seeds, Variance and Statistics

Learning objectives

After this lesson, you should be able to:

• Explain why the training seed is the unit of replication, and why pooling evaluation episodes across seeds gives an interval that is far too narrow.
• Compute a t-interval, a percentile bootstrap interval and a Wilson interval for a seed-level result and say what each one assumes and where it fails.
• Run and read a two-sided permutation test, report an effect size, and correct a set of comparisons with the Bonferroni and Holm rules.
• Say how many seeds a comparison needs, and why reporting the best seed misleads.

The intuition

A trained policy is a number you can measure: run it from 100 start positions and count successes, and the Wilson interval says how sure you can be of that number for that policy. But training PPO again with a different seed gives a different policy. The figure shows three PPO variants of lesson M11-T1, 20 seeds each, one dot per seed. Default PPO's dots pile up at the ends: ten at 0, the others at 0.52, 0.82, 0.97, 0.99 and six at 1.00. The same hyperparameters, the same number of training episodes, the same evaluation starts, and two kinds of outcome. Variation of this kind is the usual condition of deep reinforcement learning, not a defect of this toy.

Under every row of dots are two intervals for the same mean. The orange one pools all 2000 evaluation episodes (20 seeds times 100 starts) and treats them as 2000 independent coin flips: it is 0.04 wide and says 0.44 to 0.49. The blue one treats the 20 seed-level scores as the sample and says 0.24 to 0.69. The orange interval is right about something: the success rate of those twenty policies as a group. It says nothing about how a twenty-first training run would go, which is the question. Episodes from one policy are repeated measurements of that policy; seeds are replications of the method. The next two sections build the machinery to keep them apart.

The mathematics

Seeds as the sample

Train the method with seeds 1 to n and score each resulting policy on a fixed evaluation set. This gives n numbers x₁, …, x_n, one per run. Treat them as independent draws from the distribution of scores the method produces. The mean, the sample standard deviation (with n − 1 in the denominator) and the standard error are:

Each x_i also carries evaluation noise from scoring on m episodes, roughly p_i(1 − p_i)/m in variance. The total is:

The first term does not shrink as you add evaluation episodes, the second does. When the first dominates, as it does for the PPO results here, more evaluation episodes cannot make the comparison more reliable: you need more seeds.

Three intervals for the mean over seeds

• t-interval. mean ± t_n−1 · se, with t_n−1 the 97.5% point of Student's t distribution (2.776 for 5 seeds, 2.262 for 10, 2.093 for 20). It assumes the sample mean is roughly normal, which is reasonable for large n or smooth scores and doubtful for a few bimodal scores. It can cross the bounds 0 and 1.
• Percentile bootstrap. Resample the n scores with replacement B times, compute the mean of each resample, and take the 2.5% and 97.5% quantiles. It stays inside [0, 1] and needs no normality, but it only knows the values it saw: for very few seeds it is too narrow.
• Wilson interval over pooled episodes. A binomial interval for k successes in N episodes. It is correct for one fixed policy and wrong as a statement about the method, because it assumes the N episodes are independent draws from one success probability.

Permutation test, effect size, multiple comparisons

To ask whether method A differs from method B, take the two samples of seed scores, the statistic T = |mean_A − mean_B|, and ask how often a random relabelling of the pooled scores gives a statistic at least as large. If the labels did not matter (the null hypothesis), every relabelling is equally likely:

The +1 counts the observed split itself, so p is never 0. With all C(2n, n) relabellings enumerated the test is exact, and the smallest attainable two-sided p, when the groups separate completely, is 2/C(2n, n). The p-value is not the probability that the null is true, and not the size of an effect; the effect size is the difference of means with an interval, or the standardised version

Running m tests, each at level α, makes at least one false positive likely: if the tests were independent, P(at least one) = 1 − (1 − α)^m. Bonferroni multiplies each p-value by m (equivalently uses α/m). Holm sorts the p-values, multiplies the smallest by m, the next by m − 1 and so on (keeping the sequence non-decreasing), which controls the same error and rejects at least as many hypotheses.

n, m

Number of training seeds per method; number of evaluation episodes per seed (m = 100 here).

x_i

Success rate of the policy trained with seed i on the shared evaluation starts.

se

Standard error of the mean over seeds, s/√n.

B

Number of random relabellings (permutation test) or resamples (bootstrap).

d

Standardised effect size; |d| near 0.2 is conventionally called small, 0.5 medium, 0.8 large (Cohen), a rule of thumb and not a law.

A worked example

Numbers come from Node scripts against RLCore (scratch prototypes). Each variant was trained for seeds 1 to 20 with RLCore.trainPolicyGradient({algo: 'ppo', iters: 60, eps: 12, seed: s}, rng(s)) (changes for the variants as in lesson M11-T1), the mean policy was scored on the same 100 starts RLCore.starts(900, 100, 'all'), and the result is one success rate per seed. Statistics use RLCore.meanCI, bootstrapCI (5000 resamples), wilson and permutationTest (10000 shuffles).

By hand with five seeds

Take the first five default seeds: 0.97, 0.82, 0.00, 1.00, 1.00. The mean is 3.79/5 = 0.758. The deviations are 0.212, 0.062, −0.758, 0.242, 0.242, with squares 0.0449, 0.0038, 0.5746, 0.0586, 0.0586, summing to 0.7405. Then s = √(0.7405/4) = 0.4303 and se = 0.4303/√5 = 0.1924. With t₄ = 2.776 the half-width is 0.534, so the t-interval is 0.224 to 1.292. A success rate of 1.29 is impossible: with five bimodal scores the normal approximation has failed. The percentile bootstrap of the same five numbers gives 0.364 to 0.994, inside the bounds, but with five values it cannot know about the seeds it has not seen.

Twenty seeds, three intervals

The two seed-level intervals roughly agree at 20 seeds; the pooled Wilson interval is 10 times narrower. The reason is in the variance decomposition: for the default variant the variance of the seed scores is 0.2389 (std 0.489) and the average evaluation-noise term, mean p(1 − p)/100, is 0.00022, about 1/1100 of it. All the uncertainty is in the seeds. For λ = 0 the t-interval dips below 0, which a success rate cannot, and the bootstrap interval (0.00 to 0.21) is the more sensible one. Reading the 0.44 to 0.49 as "PPO scores 0.47 ± 0.02" would be an error by a factor of about ten.

Permutation tests

By hand, 4 seeds each: default seeds 1 to 4 (0.97, 0.82, 0.00, 1.00) against λ = 0 seeds 1 to 4 (0, 0, 0, 0). The observed difference is 2.79/4 = 0.6975. The pooled eight scores contain five zeros and three non-zeros, so a split reaches a difference of 0.6975 or more (in absolute value) only when all three non-zeros land in the same group: for group A that is C(5, 1) = 5 splits, and the same for group B, 10 of the C(8, 4) = 70 splits. So the exact p is 10/70 = 0.143. Four seeds cannot reach 0.05 even in this nearly perfect case: the smallest possible p with 4 seeds each is 2/70 = 0.029, and with 3 seeds each it is 2/20 = 0.10, so a three-seed comparison can never reach p < 0.05 by this test. Shuffling 20000 times instead of enumerating gives 0.1417, as it should.

Now all 20 seeds. Each variant against default PPO (difference of means, two-sided permutation p with 10000 shuffles, Cohen's d):

Two points. Only λ = 0 survives the correction for five tests (0.0028 × 5 = 0.014). And "not shown" is not "no difference": the no-clipping interval for the difference runs from -0.13 to +0.46, which contains no effect, a large benefit and a modest harm. The same 20 seeds compared pairwise across all six variants give 15 tests, of which 5 have uncorrected p < 0.05; if 15 independent tests were all true nulls, the chance of at least one p < 0.05 would be 1 − 0.95¹⁵ = 0.54, so without a correction one or two such hits are not a finding. One more detail: the same seed number across variants shares training starts and, in part, evaluation conditions, so the unpaired test is conservative or optimistic in ways I did not measure.

How many seeds, and what the best seed says

To see how a small seed count behaves, I resampled seeds with replacement from the 20 default scores (a bootstrap-style approximation, not 2000 fresh training runs). The mean of 3 seeds had a 5%-to-95% range of 0.00 to 0.99; of 5 seeds 0.10 to 0.80; of 10 seeds 0.20 to 0.70; of 20 seeds 0.28 to 0.64. The median t-interval half-width was 1.41, 0.62, 0.34 and 0.23. For a true difference of +0.16 (no clipping against default), the chance that the sample mean of the better variant came out higher was 0.61 with 3 seeds, 0.68 with 5 and 0.77 with 10; for the larger difference between default and λ = 0 it was 0.84 with 3 seeds and 0.94 with 5. A comparison with few seeds is close to a coin toss unless the effect is big.

Best-seed reporting: the mean of the best of k default seeds, drawn from the 20, was 0.48 for k = 1, 0.85 for k = 3, 0.95 for k = 5 and 1.00 for k = 10. The best seed of the 20 is 1.00. A table that reports it describes the method as a policy that reaches the goal every time, when the average run does it 47% of the time and half the runs never do. If "best of 5 seeds" for method A is set beside the mean of 5 for method B, the winner is the method with the more variable outcomes.

Finally, a rough sanity check of the test. Splitting the 20 default seeds at random into two groups of 10 and testing them against each other should rarely show a difference. Over 300 random splits the permutation test gave p < 0.05 in 7 (2.3%), and a t statistic with critical value 2.10 rejected in 20 (6.7%). The halves are not independent samples from the population (they partition one finite set of 20), so I read this only as a rough check that nothing is badly off, not as a calibrated error rate.

Engineering implications and limitations

A protocol for comparing learned policies

• Fix the seed count before looking. Choose n from a pilot (the spread of 5 to 10 seeds), the smallest effect you care about, and a budget. Ten seeds is a floor for noisy learners, not a standard; with rates that are bimodal, 20 or more is more honest.
• Score every seed on the same evaluation set, disjoint from anything used for tuning, and keep the seed-level scores, not only their mean.
• Report the distribution: the value of every seed or a strip plot, the mean or median, and a seed-level interval (bootstrap when the scores are bounded or skewed, t when they are well behaved). Never an interval over pooled episodes as the uncertainty of a method.
• Report the effect with its interval and the test, then the sentence the evidence supports: "default PPO is higher than λ = 0 by 0.38 (interval 0.14 to 0.61), permutation p = 0.003 after correction". Separate "no difference shown" from "no difference".
• Count your comparisons. Decide which are confirmatory before looking; correct them with Holm; label the rest exploratory.
• Prefer robust summaries when outcomes are bimodal. The mean of a bimodal score describes no run. Report the fraction of seeds above a stated threshold, the median and the interquartile mean; Agarwal et al. recommend interval estimates, performance profiles and the interquartile mean for deep reinforcement learning benchmarks.
• A statistical difference is not an important one. A tiny difference can reach p < 0.05 with enough seeds. State the smallest effect that would matter, and check the interval against it (lab M11-L1 combines both in one verdict rule).

Common failure modes

What this toy cannot tell you

The seeds here differ in network initialisation, exploration noise and training start positions, while the world and evaluation set are fixed; on a robot, hardware variation, calibration and wear add variance that no seed controls. The evaluation set is 100 starts of one task, so the scores carry their own sampling error and the same evaluation starts are reused by all variants. The seed-level scores are bounded and bimodal, so normal-theory intervals are doubtful and my A/A check is only a sanity check. For deep reinforcement learning on larger benchmarks, how many seeds are needed depends on the algorithm and the environment and is itself a research question (Colas et al.). No real robot data is used in this lesson.

Common misconceptions

Engineering takeaways

1. The training seed is the unit of replication: the interval for a method comes from the spread of seed-level scores, not from pooling evaluation episodes. For default PPO the seed-level t-interval was 0.24 to 0.69 and the pooled Wilson interval 0.44 to 0.49.
2. PPO on the Scout reach task was bimodal across 20 seeds (ten at 0, the rest from 0.52 to 1.00), so the mean describes no run: report the distribution of seeds, a bootstrap or t-interval, and the fraction above a threshold.
3. A permutation test with the +1 correction needs enough seeds to be capable of a small p: the minimum two-sided p is 0.10 with 3 seeds per group, 0.029 with 4 and 0.008 with 5.
4. Report effect sizes with intervals, correct for the number of comparisons (Holm), and separate "no difference shown" from "no difference". Of five PPO ablations only λ = 0 survived (p = 0.0028, adjusted 0.014).
5. Best-seed reporting inflates the result: the best of 5 default seeds averages 0.95 against a method mean of 0.47. This is a toy study on the simulated Scout, with no real robot data.

References and further reading

The intervals, the permutation test and the Holm correction are standard statistics; the numbers are from my own scratch experiments on the toy Scout world, not from the papers below, which study seed variance and evaluation practice in deep reinforcement learning on standard benchmarks.

• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning that Matters”, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
• R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville and M. G. Bellemare, “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, Advances in Neural Information Processing Systems (NeurIPS), 2021.
• C. Colas, O. Sigaud and P.-Y. Oudeyer, “How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments”, arXiv:1806.08295, 2018.
• B. Efron, “Bootstrap Methods: Another Look at the Jackknife”, The Annals of Statistics 7(1), 1979.
• S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure”, Scandinavian Journal of Statistics 6(2), 1979.
• E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference”, Journal of the American Statistical Association 22(158), 1927.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Lesson M11-T3 asks what else is needed so that a number like these can be produced again: the code version, the exact configuration and seeds, and a record of the data. I test exactly how deterministic this repository's seeded runs are.

Related content

Nothing below is required to finish this lesson.

• Before
  M11-T1: Baselines and Ablations

  Choose baselines that could win, and design ablations that show which part of a method matters.

• Lab
  M11-L1: Compare Two Policies Statistically

  Two policies, ten seeds each: decide whether one is better, with intervals and a permutation test.