M11-T4 — Writing an Experiment Report

Learning objectives

After this lesson, you should be able to:

• Structure a report as question, setup, results, failures and limits, and say what each part must contain so that a stranger could repeat the run.
• Report a result with its sample size, an interval at the right level (seeds, not episodes), the weakest region and the failure mode, not only a mean.
• Rewrite a sentence that claims more than the data supports into one the interval and the setup actually justify.
• Report a negative or inconclusive result so that it is useful: what was tried, what was found, and which effects the data cannot exclude.

The intuition

Take one sentence from the lab notebook of a fictional robot-learning project at Meridian General Hospital: "Pillar-aware features solved the reach task: 100% success." It is not a lie. Scout reached the goal on every one of 1,000 hidden starts in the easy region, where the start is far from the line through the pillar. It is also the wrong summary. In the near region, starts within one metre of that line, the same policy family reached the goal 794 times in 1,000, and its ten trained copies scored anywhere from 0.49 to 1.00. A reader who sees only the sentence will deploy into the hall where the pillar is, not the empty half of it.

The figure is the report's most honest piece of evidence: one dot per demonstration seed in the weak region, with the mean and its 95% t-interval. Three configurations are on it. Direction-only features fail near the pillar (mean 0.19). Pillar-aware features with five demonstrations reach 0.79, but the dots spread from 0.49 to 1.00 so the interval is wide (0.66 to 0.93) and the bar of 0.90 is not cleared. And a third configuration, forty demonstrations from the easy region only, is the negative result a good report includes: more of the wrong data made the near region worse.

A report exists to make that picture, in words, impossible to miss. It leads with the claim the data supports, it shows the sample and the spread, and it puts the weakest region in the same paragraph as the headline rather than in an appendix.

The mathematics

Five parts

A short experiment report has five parts, each of which answers one question a sceptical reader will ask. Write them in this order, even if you discovered them in another.

Match the sentence to the interval

The statistics of lessons M1-T5 and M11-T2 are used in a report in one way: they set how strong the sentence may be. Suppose the training run is repeated for s seeds (here the seed picks the demonstrations), each scored on the same n hidden starts, giving per-seed success rates p₁ … p_s. The headline is their mean and a Student-t interval over seeds, not a Wilson interval over the pooled s·n episodes, because the episodes of one trained policy are not independent samples of the training procedure:

s

Number of independent training runs (seeds). The interval is about the procedure, so s, not the number of test episodes, sets its width.

n

Hidden test starts per run. It sets the noise in each p_i (a Wilson interval for one run), not the run-to-run spread.

t(s-1)

Two-sided 95% Student-t critical value: 2.262 for s = 10, 2.776 for s = 5, 4.303 for s = 3.

claim range

The set of values the sentence asserts, for example "at least 0.90". The sentence is supported if the whole interval lies inside it.

That gives a rule you can apply to any sentence. Write down the quantity the sentence is about (a mean over seeds, a worst seed, a single region) and the range it asserts. If the interval for that quantity sits entirely inside the range, the claim is supported at that level. If the interval straddles the boundary, say "not distinguished from" or "between a and b". If the claim is about the worst case ("never fails"), a mean says nothing: you need the minimum over seeds and regions, and a count of how many you looked at.

Reporting a comparison

For two configurations run on the same seeds, report the difference with an interval or a permutation p-value (lab M11-L1) and its size in the units of the task. A p-value of 0.0002 says the difference is unlikely to be chance; it does not say it is large enough to matter. When several regions or metrics are tested, say how many, because the chance of at least one small p among m independent tests at level α is 1 − (1 − α)^m (0.14 for m = 3, α = 0.05).

Negative and inconclusive results

A negative result says "we tried X and Y did not improve". It is useful only if it says how hard X was tried and how large an effect the data could have detected. An interval for the difference does this: if the paired difference is 0.014 with interval −0.053 to +0.081, the data exclude a loss or a gain of more than about 0.08, and cannot exclude one of 0.05. "No significant difference" says none of that.

A worked example

Every number comes from running RLCore in Node on Scout's reach task. The study asks one question: does a pillar-aware feature set fix the near-pillar failures of direction-only behaviour cloning? Policies are linear in features: featDirection (3 features: the unit vector to the goal and a bias) against featPillar (7 features: adds the pillar offset, a closeness weight and the side of the pillar, lesson M3-T2). Training data is the expert's trajectories from 5 start positions drawn from the whole start region (starts(seed, 5, 'all'), about 150 state-action pairs). The fit is ridge regression with λ = 0.01, set before looking at results. Ten seeds (1 to 10) select the demonstration starts. The test set is the same for every run: starts(900, 100, region) for each of three regions, easy, near and all, 100 hidden starts each, never used in training. Success means reaching the goal within 80 steps without touching the pillar. The expert itself scores 100 of 100 on the 100 hidden starts of the all region.

The report (written from the run)

1. Question. Does adding pillar-aware features to behaviour cloning raise success near the pillar line to at least 0.90 with 5 demonstrations? (Criterion decided before the run: mean over seeds with the whole 95% interval at or above 0.90.)

2. Setup. RLCore reach task as above (10 m by 6 m hall, pillar at (5, 3), no sensor noise, no delay). Two policies, direction features and pillar features, each trained on the expert's actions from 5 demonstration starts; 10 seeds; ridge λ = 0.01 (λ chosen a priori; sensitivity checked after, see Limits). Hidden test: 100 starts per region, the same starts for every seed and both policies. Metric: success rate per seed and region; headline is the mean over the 10 seeds with a 95% t-interval (df = 9, t = 2.262).

3. Results.

Permutation p-values are from 10,000 shuffles of the pooled per-seed rates (two-sided, with the +1 correction, so 0.0001 is the smallest value that can appear). Pillar features beat direction features in all three regions, by 0.27, 0.61 and 0.49. The criterion of lesson M11-T1 is only met in the easy region: the headline mean of 0.91 for all starts hides that the interval for near starts, 0.66 to 0.93, does not clear 0.90.

4. Failures. The weakest region is near. Of the 206 failed near-pillar episodes (10 seeds, 1,000 episodes, 794 successes), all 206 were crashes into the pillar and none timed out. Per seed, the near-pillar success was 0.97, 0.61, 1.00, 0.74, 0.73, 1.00, 0.69, 0.71, 1.00 and 0.49. Three seeds were perfect and seed 10 scored 0.49; seed 10's five demonstration starts contained none within 1 m of the pillar line. That is suggestive and not a proof: seeds 6 and 9 each had one demonstration start within 1 m and scored 1.00, while seed 7 had two and scored 0.69. Failures are not concentrated at the very centre of the region (17% of episodes failed from starts within 0.5 m of the line, 24% from starts 0.5 to 1 m away), so the cause is not simply "closer is harder".

5. Limits. (a) Toy study: a point robot, a single pillar, no sensor noise or delay, no real robot data. (b) Ten seeds and one test set of 100 starts per region; the same 100 starts are used for all seeds, so the interval covers the variation caused by the demonstrations and not the variation of the test starts. (c) λ = 0.01 was fixed in advance; the same study with λ = 0 gives a near-pillar mean of 0.81 and with λ = 1 gives 0.64 (mean over the same 10 seeds), so the headline depends on a hyperparameter I did not tune and did not report a sweep for. (d) Only 5 demonstrations per seed were tried for this comparison; with 20 demonstrations drawn from the whole region the pillar-feature policy scored 1.00 in all 10 seeds, so the conclusion is about a small-data regime. (e) The expert is a hand-written controller, which already scores 100 of 100; the study shows what the policy learns from it, not that learning is needed (the last section of lesson M11-T5).

By hand: the interval and the permutation p

Pillar features, near region: mean 0.794, standard deviation 0.1854 over the ten seed rates. The standard error is 0.1854/√10 = 0.0586, the half-width is 2.262 · 0.0586 = 0.1326, so the interval is 0.661 to 0.927, as in the table. For the permutation test the observed absolute difference of means is 0.794 − 0.188 = 0.606. Pool the twenty per-seed rates, shuffle them 10,000 times into two groups of ten and count shuffles whose absolute difference is at least 0.606: 1 of 10,000, so p = (1 + 1)/(10,000 + 1) = 0.0002. The Wilson interval for the pooled 794 of 1,000 episodes is 0.768 to 0.818, about a fifth as wide as the seed-level interval: it would be the wrong interval for a claim about the training procedure, because it treats the ten policies' episodes as one policy's.

Before and after: an overclaiming sentence

Common repairs: replace "solves" and "robust" by the number and the condition; "outperforms" by the difference and its interval; "always" and "never" by a count over a stated number of trials (200 successes in 200 trials is a Wilson lower bound of 0.98, not "100%"); and the headline mean by the mean and the weakest region.

A negative result, written out

The third row of the figure came from a question the project asked next: if five demonstrations leave a gap near the pillar, does collecting more of the cheap demonstrations help? Forty demonstrations from the easy region only, pillar features, same ten seeds and test starts: near-pillar success fell from 0.30 (5 easy-region demonstrations; interval 0.20 to 0.40) to 0.05 (40 of them; interval 0.02 to 0.07), p = 0.0001. The report entry reads: "We expected more easy-region demonstrations to help near the pillar. They did not: 8 times more data lowered near-pillar success by 0.25 (0.30 to 0.05, ten seeds). The extra data contain almost no examples of the manoeuvre the weak region needs. This tests data from the easy region only; we did not test 40 demonstrations drawn from the whole region (20 of those already reach 1.00 in all ten seeds)."

A second negative result is the null: λ = 0 against λ = 0.01 on the near region, per-seed differences with mean 0.014, standard deviation 0.094 and a paired 95% interval of −0.053 to +0.081 (unpaired permutation p = 0.90). The honest entry is: "No difference detected; differences of more than about 0.08 in either direction are unlikely with this sample; a difference of 0.05 is not excluded." Compare it with "λ does not matter", which the data cannot support: λ = 1 gave 0.64 and λ = 10 gave 0.18.

Engineering implications and limitations

Writing practice

• Write the question and the criterion before the run. If the success criterion is written afterwards it will drift to wherever the result landed. A one-line entry in the experiment log (lesson M11-T3) with a date is enough; lesson M11-T5 turns this into a pre-registration checklist.
• Put the number, the sample and the interval in the same sentence. "0.79 (0.66 to 0.93, 10 seeds)" cannot be quoted without its uncertainty.
• Report every region or condition you tested. If you tested three and show one, say so. A hidden comparison is the garden of forking paths: with three tests at α = 0.05 the chance of at least one false positive is already 14%.
• Name the worst seed and the worst region next to the mean. The mean is an average over runs you might never deploy; one run goes to the robot.
• Say which numbers were tuned on the test set. If you looked at the hidden starts while choosing λ, they are validation data and the report needs a fresh test.
• Give failures a mechanism or say you have none. "All 206 failures were collisions" is a finding. "The cause is the missing demonstrations" is a hypothesis unless you tested it (one seed with no near-line demonstration scored lowest, but seeds with one scored both 1.00 and 0.69).
• Keep the limits specific. "Simulation may differ from reality" is true of every report; "no sensor noise or delay was simulated, and lesson M10-T4 shows delay alone moved a policy from 1.00 to 0.44" is a limit.

What this report does not tell you

The report is for a simulated hall with a point robot, an expert that already succeeds, and a test set the author chose. A checklist makes a report honest, not the study large: the 100 starts per region sample a start distribution I defined, not any hospital's, and a test set that leaks into training (lesson M2-T6) makes the best-written report a well-written mistake.

Common misconceptions

Engineering takeaways

1. A report has five parts: the question and criterion, the setup, the results with intervals, the failures, and the limits. Each answers a question a sceptical reader will ask.
2. Size the sentence to the interval: in the Scout study pillar features raised mean success from 0.41 to 0.91 over ten runs, but near the pillar line the mean was 0.79 (0.66 to 0.93), so "solves the task" is not supported.
3. Report the weakest region and the worst seed next to the mean, and the failure mode: all 206 near-pillar failures were collisions, and one run scored 0.49.
4. Use seeds, not episodes, for the interval about a training procedure: the pooled Wilson interval (0.77 to 0.82) was about five times narrower than the t-interval over runs (0.66 to 0.93).
5. Report negative and null results with the effect size the data can exclude: 40 easy-region demonstrations lowered near-pillar success from 0.30 to 0.05; λ = 0 against 0.01 differed by 0.014 (−0.053 to +0.081). This is a toy study on the simulated Scout, with no real robot data.

References and further reading

The five-part structure is a standard scientific-report pattern adapted to robot learning; the numbers are from my own scratch runs on RLCore, not from a paper. The cited works are general references on reproducibility and documentation, not sources for the Scout numbers.

• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning That Matters”, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
• M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji and T. Gebru, “Model Cards for Model Reporting”, Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), 2019.
• E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference”, Journal of the American Statistical Association 22(158), 1927.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A report describes an experiment that already happened. Lesson M11-T5 moves to the beginning: how to turn a task such as "deliver this item across the ward" into a learning problem with data, a simulator, a metric, hidden test scenarios, a budget and a safety plan, written down before any training run.

Related content

Nothing below is required to finish this lesson.

• Before
  M11-T3: Reproducibility and Experiment Tracking

  Record code version, seeds, configuration and data so that a result can be repeated.

• Lab
  M11-L2: The Learning Capstone

  Train, evaluate and diagnose a policy for Scout on hidden scenarios, then write the experiment report.