LM12-T2 — Experiment Design and Reproducibility

Learning objectives

After this lesson, you should be able to:

• Design a controlled comparison: one question, baselines, one changed factor.
• Explain what a seed controls and why several seeds are needed.
• Compute a t-interval across seeds and estimate how many runs a conclusion needs.
• Log configurations, environments and results so that a run can be reproduced.

The idea: separate the effect from the noise

Suppose masked training scores 0.82 and unmasked 0.79 on a held-out slice. Is masking better? That depends on how much a single method varies from run to run. If rerunning the same method with another seed can move the score by 0.2, a difference of 0.03 means nothing. If the spread is 0.01, it is something. An experiment is designed so that the noise can be measured and the effect can be compared against it.

The anatomy of a controlled comparison

1. A precise question,“Does masking the prompt in the loss change exact match on unseen bay numbers?”, not “is masking good?”.
2. Baselines,Trivial and competent comparators, using only training data (Notebook LM12-L2). They give the scale of a result.
3. One changed factor,Variants differ in exactly one thing: same data, same split, same model size, same steps, same learning-rate schedule, same evaluation.
4. Several seeds,Repeat every variant with different seeds; report each run and the summary.
5. Predeclared metrics,Metric, test set and decision rule fixed before looking at results.
6. A record,Configuration, code version, environment, seed, time and results saved per run.

What a seed controls

A seed initialises a pseudo-random generator. In one training run, different seeds can govern the initial weights, the order in which examples are drawn, dropout masks and any data augmentation. Each source matters: the initial weights can decide whether a run falls into a good solution; the order can decide which examples the model sees last. In the capstone notebooks the model seed and the batch order are both set by the run’s seed, so a seed is a bundle of choices. A fixed seed on CPU gives exactly the same loss curve twice (Self-check 2 of Notebook LM12-L2); on a GPU some operations are non-deterministic unless you ask for deterministic kernels, and results can differ by small amounts across hardware and library versions.

Ablations

An ablation removes or changes one component to see what it contributes. The skill is in the discipline: when you change the learning rate and the masking and the number of steps together, you cannot say which mattered. Change one thing, keep the rest identical, and give both variants the same compute. Interactions (masking may help only for a larger model) need separate experiments; say so as a limitation rather than guessing.

Held-out data are used once

Development decisions (learning rate, number of steps, which checkpoint) use the validation split. The test split is looked at once, at the end. Every look at the test set that changes a decision turns it into another validation set, and the reported number becomes optimistic. If you have to look, say so (Lab LM12-L4 Stage 1 shows the verdict).

Report negative and null results

The mathematics: intervals across seeds and the number of runs

Let a method have true mean performance μ and run-to-run standard deviation σ. With n independent runs giving scores x₁ … x_n:

95% interval for μ:

t(1) = 12.706

two runs; t(2) = 4.303 (three runs); t(4) = 2.776 (five runs); t(9) = 2.262 (ten runs); t → 1.96 for many runs.

s

estimated from the same runs; with two runs it is a poor estimate, so the interval is very wide.

Worked numbers. The two unmasked runs of the capstone reached 0.787 and 0.551 on the held-out slice: x̄ = 0.669, s = 0.167, SE = 0.118, and the 95% interval is 0.669 ± 12.706 · 0.118 = 0.669 ± 1.499, which covers every possible score. Two seeds cannot establish the level of a method with that much variation. With five runs 0.80, 0.78, 0.83, 0.55, 0.79 the interval is 0.75 ± 0.141 (one bad run); with five stable runs 0.80, 0.78, 0.83, 0.79, 0.81 it is 0.802 ± 0.024.

How many runs?

To estimate a mean to within a half-width h when the run-to-run standard deviation is σ, you need about

For σ = 0.11 and h = 0.05: (2.776 · 0.11 / 0.05)² = 38 runs. This is why small differences in noisy settings cannot be resolved with a laptop budget; either reduce the noise (a larger evaluation set, a more stable training recipe, paired comparisons on the same examples) or claim less.

Two kinds of variation

The scores above vary for two reasons. Training noise (seeds) changes the model; evaluation noise (a finite test set) changes the measured score of a fixed model. The Wilson interval of Module 9 addresses the second, the interval across seeds the first. A report needs both: for example, report the per-seed scores, their mean and spread, and, for the main model, the Wilson interval on the test set.

Reproducibility

A run is reproducible if someone with your repository can obtain the same result. That needs the code at a fixed version, the exact configuration (hash it, so equal settings have equal identifiers), the seed, the environment (Python and library versions, hardware), the data version or the generator seed, and the commands to run. Floating-point results can still differ by small amounts across hardware; state the tolerance.

A worked example: intervals, run counts and configuration ids

The code reproduces the interval arithmetic above, a configuration hash that does not depend on the key order, and the run-count formula.

import hashlib
import json
import math

def mean_sd(values):
    m = sum(values) / len(values)
    sd = math.sqrt(sum((v - m) ** 2 for v in values) / (len(values) - 1))
    return m, sd

T975 = {1: 12.706, 2: 4.303, 3: 3.182, 4: 2.776, 9: 2.262}      # two-sided 95% t quantiles by degrees of freedom

def interval(values):
    m, sd = mean_sd(values)
    half = T975[len(values) - 1] * sd / math.sqrt(len(values))
    return round(m, 3), round(half, 3)

print(interval([0.787, 0.551]))                      # (0.669, 1.499)  two seeds: the interval is useless
print(interval([0.80, 0.78, 0.83, 0.55, 0.79]))      # (0.75, 0.141)   five seeds: still wide because one run is an outlier
print(interval([0.80, 0.78, 0.83, 0.79, 0.81]))      # (0.802, 0.024)  five stable seeds

def config_id(config):
    blob = json.dumps(config, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:8]

a = {"mask_prompt": True, "steps": 1000, "d": 64, "layers": 2, "seed": 0}
b = {"seed": 0, "layers": 2, "d": 64, "steps": 1000, "mask_prompt": True}
c = dict(a, seed=1)
print(config_id(a) == config_id(b), config_id(a) == config_id(c))    # True False  (same settings, same id, whatever the key order)

def runs_needed(sd, half_width, t=2.776):
    """Smallest n with t * sd / sqrt(n) <= half_width (t fixed for about 5 runs; refine if n changes much)."""
    return math.ceil((t * sd / half_width) ** 2)

print(runs_needed(0.11, 0.05))                       # 38

The last two numbers carry the message of the lesson. The first interval, 0.669 ± 1.499, says nothing at all. The run-count formula says that resolving a half-width of 0.05 at a standard deviation of 0.11 would need about 38 runs per variant, which is out of reach for most capstones; the honest response is to report what you have with its limits, or to design a less noisy comparison.

The run record

Engineering implications and limitations

Seed everything and record it. Python’s random, NumPy, PyTorch (and its CUDA generators), and the data-loader workers. Write the seed into the run record.

Create generators once. A batch generator seeded before the loop gives the same batches every time; re-seeding inside the loop gives the same batch repeatedly, a classic bug.

Use the same data order for paired variants when it makes sense. If you want to isolate the effect of a model change, giving both variants the same batches removes one source of noise; if you want the total variability of a method, vary everything.

Pin the environment. A requirements file with exact versions, and the output of the version commands in the run record. Expect small numerical differences when the hardware changes.

Automate the table. Produce the results table from the run records by a script, never by copying numbers by hand.

Keep a failed-runs log. Diverged runs and bugs belong in a note, with the fix; they explain choices and prevent others from repeating them.

For robots. If your results will inform a physical system, repeat the key evaluation on the target hardware; timing and numerics differ, and the conclusion “it works” is about the system you tested.

Common misconceptions

Engineering takeaways

1. A comparison needs a precise question, baselines, one changed factor, several seeds, a predeclared metric and a run record.
2. A seed bundles several random choices; fixed seeds reproduce runs exactly on CPU in a fixed environment.
3. Interval across n seeds: x̄ ± t(n−1)·s/√n. Two seeds give t = 12.706: 0.669 ± 1.499 in the capstone example. Runs needed ≈ (t·σ/h)².
4. Report training noise (across seeds) and evaluation noise (Wilson interval) separately.
5. Hash the configuration, store the environment and generate result tables by script.

References and further reading

The quantiles of Student’s t used above are standard table values. The accuracy values for five seeds in the worked example are made-up numbers chosen to illustrate the arithmetic; only the two unmasked scores (0.787 and 0.551) come from the capstone notebook run. See Module 9 for confidence intervals and paired tests.

• Bouthillier, X. et al. (2021). Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems.
• Pineau, J. et al. (2021). Improving reproducibility in machine learning research. Journal of Machine Learning Research 22.
• Student (Gosset, W. S.) (1908). The probable error of a mean. Biometrika 6(1).
• Build a Large Language Model (From Scratch), the whole book (primary book reference)
• Publisher page for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: evaluating and presenting your system: held-out data, slices, errors and a demonstration.

Related content

Nothing below is required to finish this lesson.

• Before
  LM12-T1: Capstone Brief and Scoping for Limited Compute

  Choose a feasible project and define milestones and success criteria.

• Lab
  LM12-L2: Capstone Milestone 2: Model and Training Experiment

  Train or fine-tune a model with a baseline, an ablation and logged results.