LM9-T3 — Sampling Variability and Reproducibility

Learning objectives

After this lesson, you should be able to:

• List the sources of randomness in training and evaluation of a language model.
• Compute the mean, standard deviation and a t-based interval over seeds, and a bootstrap interval over items.
• Distinguish variation from the seed from variation due to the finite evaluation set.
• Make a run reproducible, and say what reproducibility can and cannot guarantee.

The idea: a result is a draw from a distribution

Each time you train or evaluate, random choices are made. Treat the number you get as one draw from the distribution of possible results. To report it honestly, estimate the width of that distribution.

Sources of randomness

Two kinds of variance

Seed variance: the same system, different random choices, same evaluation items. Item variance: the same system, a different sample of evaluation items. Both matter. Greedy decoding on a fixed, trained model removes seed variance at evaluation but not item variance. When you compare two trained models, training seeds add variance of their own: two training runs with different seeds of the same recipe can differ by more than the effect of a small change of recipe.

How many seeds?

Three to five seeds give a rough spread; ten give a usable interval. With fewer than three you cannot estimate the spread at all. For expensive training runs, you may only afford one or two seeds; then use the paired comparison on the same items to reduce noise, and be modest in your claims.

Reproducibility

A run is reproducible if someone with your record can obtain the same result. That needs code version, data version, configuration, seeds, library and hardware versions. Bit-for-bit identical results are often possible on one machine with deterministic settings, and often not across machines; the realistic aim is results within the stated variation.

The mathematics: mean, spread and intervals

Let x₁, …, x_k be the metric from k seeds. The mean and the sample standard deviation are

k

number of runs (seeds).

s

the spread of a single run; it does not shrink as you add seeds.

s/√k

the standard error of the mean; it shrinks like 1/√k.

A 95% interval for the mean, with few seeds, uses the t distribution:

For five seeds with accuracies 0.717, 0.800, 0.733, 0.700, 0.750: x̄ = 0.740, s = 0.038, s/√5 = 0.017, interval 0.740 ± 2.776 × 0.017 = 0.692 to 0.788. A single run of this system lies, 95% of the time, within about ±2s = ±0.075 of the mean. A claimed improvement of 0.02 is far inside that.

Item variance. For n items and accuracy p, the standard deviation due to the finite set is √(p(1 − p)/n): 0.052 for p = 0.8 and n = 60. For the seeds above (s = 0.038) the two are of the same order. If seed and item variation are independent they add in variance: total ≈ √(0.052² + 0.038²) = 0.064 for a single run on a fresh sample of items.

Bootstrap over items. Resample the n items with replacement B times (1,000 or more), compute the metric each time, and take the 2.5% and 97.5% quantiles. For 48 correct out of 60 this gives about [0.68, 0.90]. For comparing systems, resample items jointly for both (a paired bootstrap, Lab LM9-L1).

Seeds in comparisons. To compare recipe A and B with k seeds each, compare the two means with the standard error of their difference, √(s_A²/k + s_B²/k). Using the same seeds for both (same data order and initialisation where possible) reduces the noise when the effect of the seed is shared.

A worked example: five seeds and a bootstrap

The code simulates a sampling system with true accuracy 0.8 evaluated on 60 items under five seeds, computes the mean, the spread and a t-based interval, and a bootstrap interval for one run.

import math
import random

def run_once(seed, p=0.8, n=60):
    rng = random.Random(seed)
    return sum(rng.random() < p for _ in range(n)) / n

accs = [run_once(s) for s in range(5)]
mean = sum(accs) / 5
sd = (sum((a - mean) ** 2 for a in accs) / 4) ** 0.5
print([round(a, 3) for a in accs], round(mean, 3), round(sd, 3))     # [0.717, 0.8, 0.733, 0.7, 0.75] 0.74 0.038

se = sd / math.sqrt(5)
half = 2.776 * se                                                   # t value for 4 degrees of freedom
print(round(se, 3), f"interval {mean - half:.3f} to {mean + half:.3f}")   # 0.017 interval 0.692 to 0.788

def boot_ci(results, B=1000, seed=0):
    rng = random.Random(seed)
    n = len(results)
    means = sorted(sum(rng.choice(results) for _ in range(n)) / n for _ in range(B))
    return means[int(0.025 * B)], means[int(0.975 * B) - 1]

print(boot_ci([1] * 48 + [0] * 12))                                 # (0.6833333333333333, 0.9)

print(round(math.sqrt(0.8 * 0.2 / 60), 3))                          # 0.052  spread due to the finite item set
print(round(math.sqrt(0.052 ** 2 + 0.038 ** 2), 3))                 # 0.064  both sources together

The true accuracy in this simulation is 0.8, which lies outside the seed-based interval for the mean (0.692 to 0.788) of these particular five runs: an interval at the 95% level misses the truth one time in twenty. That is a reminder that an interval is a statement about a procedure, not a guarantee about one result.

Making a run reproducible

1. Seed everything you control: Python’s random, NumPy, PyTorch (torch.manual_seed), and the data sampler (pass an explicit torch.Generator).
2. Fix the evaluation set and its order; use greedy decoding or a recorded seed per item.
3. Record versions: code revision, library versions, hardware, data and configuration.
4. Where exact repeatability matters, enable deterministic algorithms (torch.use_deterministic_algorithms(True)) and accept slower kernels.
5. Repeat important results with other seeds and report the spread.

Engineering implications and limitations

Report a spread, not a single run. “0.74 ± 0.05 (5 seeds, 60 items)” says more than “0.74”.

Use the same evaluation items for every system. A comparison on different samples confuses the effect of the model with the luck of the draw.

Use greedy decoding to remove sampling noise from comparisons, and evaluate sampling separately when sampling is how the system is used.

Do not choose the best seed. Reporting the best of ten seeds is selecting noise. Report the mean and the spread; if you must deploy one model, choose by a development set and evaluate it on the final test set once.

Watch for training instabilities. A recipe whose seeds sometimes diverge needs the divergence rate reported, not hidden by dropping failed runs.

Parallel hardware may break exact repeatability. Summation order changes results in the last bits, and a tiny difference can grow through a long generation. Expect tiny differences, and never expect bit-identity across machines.

Log seeds with outputs. When a bad sample is reported by a user, the seed and settings let you reproduce it.

For robots. A sampled plan that is right 80% of the time with a run-to-run spread of ±4 points will fail differently each time. Test with many seeds in simulation, and use constrained or deterministic decoding for the parts that must not vary.

Common misconceptions

Engineering takeaways

1. Results are draws: seeds, data order, dropout, sampling, hardware and the finite evaluation set all add variation.
2. Over k seeds report the mean and s; interval x̄ ± t·s/√k with t = 2.78 for 5 seeds and 2.26 for 10.
3. Item variance √(p(1 − p)/n) and seed variance add in quadrature: 0.052 and 0.038 give about 0.064.
4. Use a bootstrap over items (paired for comparisons); never report the best seed.
5. Reproducibility means recording code, data, configuration, seeds and versions; expect agreement within the spread, not bit-identity across machines.

References and further reading

See Chapter 5 of the book (manual seeds and deterministic behaviour in training and decoding). The statistical formulas are standard textbook results.

• Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall.
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: hallucination and prompt sensitivity: when fluent answers are unsupported, and when the wording decides the answer.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T2: Task Evaluation and Test Sets

  Design held-out test sets and metrics that match the task, and avoid peeking.

• Lab
  LM9-L1: Compare Two Checkpoints

  Compare two models with perplexity and a paired bootstrap, and decide whether the difference is real.

• Lab
  LM9-L3: Evaluate a Tiny GPT End to End

  Compare checkpoints and decoding settings on a held-out set, with seeds, confidence intervals and a latency and memory measurement.