M11-T3 — Reproducibility and Experiment Tracking

Learning objectives

After this lesson, you should be able to:

• Separate re-running exactly (same inputs, same numbers) from reproducing a conclusion (new seeds or a new implementation, same finding), and state which one a claim needs.
• List what a run record must contain: code version, the complete resolved configuration, every seed and the stream it feeds, data identity, environment, results and artifacts.
• Predict when a seeded run stays identical and when it changes, including the pitfalls of two seeds, seed 0, and call order.
• Compute a hash of a canonical configuration and use it as part of a run identifier.

The intuition

Run Scout's PPO training with seed 8 (40 iterations of 10 episodes) twice, in the same process or in different ones. Both runs end with a mean training return of 4.44, score 65 of 100 on the evaluation starts and have first network weight -0.299814589556: the same to the last printed digit. Run it with seed 9 and you get a return of 7.815, 100 of 100 and a weight of -0.257093289996. Nothing in this is magic: a seeded generator is a deterministic function that turns an integer into a stream of numbers, and the whole training run is a deterministic function of that stream.

That determinism is a gift for research, because it lets you re-run exactly, but it is also a trap. If the code, the settings, the seed or the data change in a way that was not written down, the same name produces a different number, and nobody can tell which change did it. In this repository there are two seeds in one call, a seed 0 that is secretly seed 1, and a result that changes if you draw one extra random number first; the example section shows all three. The figure is the structure of the habit that prevents these problems: four inputs recorded, one run, a record with an identifier built from the inputs.

The mathematics

A run is a function

Write the outcome of an experiment as

The run is exactly repeatable if calling F twice with the same five arguments returns the same result. It fails to be when F has a hidden input: the clock, Math.random (the unseeded generator), the order in which a container or a directory is read, the order of a floating-point sum when work is split across threads or devices, a library version, a different GPU. Exact repeatability is the strongest guarantee and the easiest to lose. Weaker, and usually the one that matters for a claim, is reproducing the conclusion: a new run with fresh seeds, or someone else's implementation, reaches the same finding within the statistical noise of lesson M11-T2. Both are called reproducibility in the literature, with the words used in different ways by different authors; in this lesson I call them re-running exactly and reproducing the conclusion.

Seeds and streams

A pseudo-random generator has an internal state a, updated by a fixed rule at every draw; the seed sets the initial state. The k-th number depends on the seed and on k:

So a result depends on how many numbers were drawn, and in what order, before the number you care about. Three consequences: (1) the same seed gives the same stream; (2) inserting one extra draw anywhere earlier shifts every later draw, so adding a print statement that draws a number can change the result; (3) two streams that share one generator are coupled, so give each independent source of randomness (initialisation, exploration noise, training starts, evaluation starts) its own stream, and derive the seeds from a master seed and a name rather than from a counter, so adding a component does not renumber the others.

The configuration hash

A configuration is a set of named settings. To get a short identifier for it, serialise it canonically (keys in sorted order, one fixed number format), feed the bytes to a hash function H and keep a prefix. The same settings in a different key order must give the same identifier, and one changed value must give a different one:

With a 32-bit identifier and N runs, the chance that some two runs share an identifier is about N²/2³³ (a birthday-problem approximation): about 1 in 8600 for N = 1000. That is fine for a notebook and too weak for a team, where 12 or more hex digits of a cryptographic hash are used. A hash identifies a configuration; it does not describe it, so the full configuration is stored beside it.

F

The experiment as a function. If it is not a function of what you recorded, it is not reproducible.

resolved configuration

The settings after defaults are filled in. Hashing only the overrides makes two runs with different defaults look identical.

canonical form

A unique text for each configuration: sorted keys, fixed formatting, no incidental whitespace.

H

A hash function: SHA-256 in practice. FNV-1a (32 bit) is small enough to do by hand and is used below only to show the mechanism.

A worked example

Every number comes from Node scripts against RLCore and SLAMCore.rng (Node v22.22.0, scratch prototypes). The PPO runs are RLCore.trainPolicyGradient({algo: 'ppo', iters: 40, eps: 10, seed: cs}, rand) with rand the next function of SLAMCore.rng(rs); "eval" is the success of the mean policy on RLCore.starts(900, 100, 'all').

Same seed, same numbers; different seed, different numbers

The generator itself: SLAMCore.rng(1) yields 0.627074, 0.002736, 0.527447 as its first three numbers every time; SLAMCore.rng(2) yields 0.734251, 0.324998, 0.285296. The evaluation starts RLCore.starts(900, 3, 'all') are [2.1801, 1.2802], [3.3608, 1.5305], [1.4523, 3.3970] on every call. On the Module 10 toy, transferScore of the tight policy on the floor world with 40 episodes scores 0.675 for seed 1 twice in a row, 0.650 for seed 2 and 0.725 for seed 3. For whole training runs:

The first two rows are identical to every digit, and the 65 of 100 for seed 8 matches a different process run earlier with the same arguments, so this is repeatability across processes on one machine. Row three changes everything: different seed, different final return, success and weight. The last three rows are the traps. (1) Two seeds. trainPolicyGradient takes a seed field in its configuration, which selects the training start positions, and a separate random stream rand supplying the network initialisation and the exploration noise. Seed (8, 9) and seed (9, 8) are different experiments from (8, 8) and (9, 9), and the returned cfg records only the first of the two. A record that lists one number as "the seed" cannot re-run the experiment. (2) Call order. Taking one extra number from the generator before training changed the same nominal run from 65 of 100 to 0 of 100. (3) Seed 0. SLAMCore.rng starts from (seed >>> 0) || 1, so seed 0 is seed 1: rng(0), rng(1), rng(1.5) and rng(4294967297) all give 0.627074, 0.002736, 0.527447. A seed sweep that runs 0, 1, 2, … trains two identical runs and counts them as two independent seeds; start at 1 and use integers.

Notice the independence by design in the other direction: RLCore.starts(seed, n, region) uses its own generator (seeded from the seed argument), so the evaluation starts do not move when training consumes more or fewer random numbers. This is the arrangement to copy: the evaluation set is a function of an evaluation seed alone.

A configuration hash by hand

FNV-1a, 32-bit: start with h = 2166136261 (hex 811C9DC5); for each byte b of the text, set h = (h XOR b) · 16777619 mod 2³². The multiplier is 0x01000193 = 2²⁴ + 0x193, so multiplying means adding h shifted left by 24 bits (low byte only survives) to h · 0x193, all mod 2³². For the text a=1, whose bytes are 0x61, 0x3D, 0x31:

Step 1 in detail: h · 0x193 = 0x811C9DA4 · 403 = 872952899884 = 0xCB400C292C, whose low 32 bits are 0x400C292C; h shifted left by 24 bits has the low byte 0xA4 in the top position, giving 0xA4000000; their sum is 0xE40C292C. So the hash of a=1 is 0x5BB13BF6, and of a=2 (last byte 0x32) 0x5AB13A63: changing one character changes the value. I checked both with a three-line program. For real use take SHA-256 of the canonical text and keep a prefix. The PPO configuration {"algo":"ppo","clip":0.2,"eps":10,"iters":40,"lam":0.95,"lr":0.003,"seed":8} serialised with sorted keys has SHA-256 prefix 965c58f2; the same settings with the keys in another order give 4a7b9343 and 34b3949e through plain JSON.stringify but 965c58f2 through the canonical form; changing lam from 0.95 to 0.9 gives b4e51f92.

Which configuration? The overrides {"algo":"ppo","eps":10,"iters":40,"seed":8} hash to 9cbd3b07, but the resolved configuration returned in cfg has 15 settings (clip 0.2, entCoef 0, epochs 5, gamma 0.99, hidden 32, initStd 0.5, lam 0.95, lr 0.003, maxGrad 0, region all, vcoef 0.5 and the four I set) and hashes to 45bcb535. If a default changes in the code, the overrides do not change but the resolved configuration, and the behaviour, do. Hash what the run used.

A run record

One line per run is enough, as long as every field is there:

{"id": "45bcb535-s8-g8", "code": "<git rev-parse HEAD> + diff hash if dirty", "config": {15 resolved settings}, "seeds": {"config.seed": 8, "generator": 8, "eval_starts": 900}, "data": "RLCore.starts(900, 100, all)", "env": "node v22.22.0", "results": {"eval_success": 0.65, "final_return": 4.44}, "artifacts": ["policy weights"]}

Five caveats about the experiment itself. First, exact repeatability here is for one machine, one runtime and one code version; I did not test other machines. The ECMAScript standard lets implementations differ slightly in functions such as Math.exp and Math.tanh, so a different engine could in principle produce slightly different weights, which a chaotic training run would amplify into a different outcome. Second, a bit-identical re-run reproduces the number, including a bad one: the seed 8 run is a failure (65 of 100) and re-running it proves nothing about the method (lesson M11-T2). Third, the record above has no commit because the code under test here is the scratch prototype: a real record needs the real hash. Fourth, I did not time-stamp or hash the data files, because the data are generated from seeds in code; real demonstration logs need a content hash. Fifth, none of this concerns real hardware, where the environment itself is not repeatable.

Engineering implications and limitations

What goes in the record

Habits

• Derive every seed from one master seed and a name (for example the hash of "init" with the master seed) so one number reproduces the experiment and adding a component does not shift the rest.
• Keep the evaluation set a function of its own seed, so changes in training cannot move the test (as RLCore.starts does).
• Never use the unseeded generator (Math.random, the clock) in anything you will report; search for it before a run.
• Write the record before the run starts and update it after, so a crashed run leaves a trace. Store failures; they are data (lesson M11-T2 on best-seed reporting).
• Check exact repeatability once: run the same configuration twice, compare the results to the last digit, and write down whether they matched. If they do not, find the hidden input before running anything else.
• Share what a stranger needs: the code at a commit, the configuration file, the command and the record; Pineau et al. propose a checklist of this kind for machine learning papers, covering code, data, hyperparameter search, evaluation and statistics.

What exact repeatability does not give you

It is necessary for debugging (you can bisect a regression) and not sufficient for a claim. A claim is reproduced when a new seed set gives the same finding within noise (lesson M11-T2), when a second implementation agrees, and, for a robot, when the result holds on the floor (lessons M10-T4 and M10-T5). Neither the hash nor the record replaces a held-out test, and a perfectly logged experiment can still be unfair (lesson M11-T1). On real hardware, repeatability is limited by the world: friction, battery state and lighting are not seeded.

Common misconceptions

Engineering takeaways

1. A run is a function of code, resolved configuration, seeds, data and environment; if any input is not recorded, the result cannot be regenerated.
2. In this repository seeded runs are exactly repeatable: PPO with seed 8 gave a return of 4.44, 65 of 100 and weight -0.299814589556 on every run, and seed 9 gave 7.815, 100 of 100 and -0.257093289996.
3. Three traps: two seeds in one training call (configuration seed and generator), seed 0 equal to seed 1, and any extra draw before training, which changed 65 of 100 to 0 of 100.
4. Hash the canonical, resolved configuration (sorted keys, defaults filled in) and build the run identifier from the code version, the configuration hash and the seeds; store the full configuration beside the hash.
5. Record every run, failures included, with per-seed results and the evaluation protocol. These are toy runs on the simulated Scout on one machine, no real robot data is used, and repeating a number is not the same as reproducing a finding.

References and further reading

The determinism results, hash values and run record are from my own scratch experiments on this repository's seeded generator and toy Scout world; I did not test other machines or runtimes. The papers below discuss reproducibility in machine learning and reinforcement learning research.

• J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivière, A. Beygelzimer, F. d'Alché-Buc, E. Fox and H. Larochelle, “Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)”, Journal of Machine Learning Research 22(164), 2021.
• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning that Matters”, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
• National Institute of Standards and Technology, “Secure Hash Standard (SHS)”, FIPS PUB 180-4, 2015.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

With the comparison designed (lesson M11-T1), the statistics done (lesson M11-T2) and the run recorded, lesson M11-T4 turns them into a document: how to state the question, the setup, the results, the failures and the limits of an experiment report.

Related content

Nothing below is required to finish this lesson.

• Before
  M11-T2: Seeds, Variance and Statistics

  Report results over several seeds with intervals, and say when a difference is larger than the noise.

• Lab
  M11-L2: The Learning Capstone

  Train, evaluate and diagnose a policy for Scout on hidden scenarios, then write the experiment report.