LM12-T4 — Writing the Report, Limitations and Next Steps

Learning objectives

After this lesson, you should be able to:

• Structure a report with the sections a reviewer expects.
• Map each claim to a table, figure or test and weaken claims that lack support.
• Write a limitations section that names untested conditions and known failures.
• Provide a reproduction guide and plan next steps.

The idea: the report is an argument with evidence

A report makes a few claims and defends each with evidence. If the claim is “the model reaches 0.82 exact match on unseen bay numbers”, the evidence is a table that shows the number, the size of the test set behind it, the interval, the baseline and the setting. If the claim is “masking is better” and the evidence is two seeds that disagree and a paired p-value of 0.61, the report must weaken the claim to what the data support: “no detectable difference with this setup”.

The sections

Tables and figures

Every table has a caption that says what is measured, on which data and with how many examples; every figure has labelled axes and units. Show per-seed values in addition to means. Use the same ordering and number formats throughout. Generate them from the run records by script (Lesson LM12-T2) so that the text and the tables cannot disagree.

Wording claims

Limitations that carry information

A useful limitations section is specific. It names conditions you did not test (other phrasings, other languages, noisy speech transcripts), failure modes you saw and did not fix (digit errors on unseen numbers), threats to validity (synthetic data, one model size, few seeds, the rule-based baseline that wins), and what would change your conclusion. It is not a confession; it is the part of the report a reader can trust most.

The sceptical-reader test

The mathematics: consistency checks you can compute

Reports contain numbers that must agree with each other. Mechanical checks catch most inconsistencies:

In the capstone: 93/97 = 0.959 and Wilson (0.899, 0.984); the slice counts must add up to 93 of 97. If a table says 96% and a figure says 94%, one of them is wrong. Compute every derived number from the saved counts by script and re-run the script when anything changes.

Effect size, not only significance

For comparisons report the counts that the test used (a_only and b_only), the difference in rates and its uncertainty, in addition to the p-value. A small p-value for a tiny difference and a large p-value for a large but noisy difference are both common; the reader needs the size to judge relevance.

Compute statement

State hardware, time and an estimate of the compute: for the default project, 1,000 steps of batch 32 × 75 tokens with 111,168 parameters, 6·N·D ≈ 1.6·10¹² FLOPs per run, about half a minute on a laptop CPU; the notebooks perform a total of about ten runs. Others can then judge the cost of reproduction and compare it with their own.

Next steps are hypotheses

Write next steps as questions with the experiment that would answer them: “Does the digit error on unseen bays shrink with a subword tokenizer? Compare character and BPE tokenizers with five seeds on the same OOD slice.” A list of vague wishes adds nothing; a list of testable questions shows where you stand.

A worked example: a mechanical report checker

The checker below tests a draft report, represented as sections, claims and a few key numbers, for the problems that most often sink a report: missing or thin sections, claims without evidence, missing baseline or sample size, and too few seeds. It does not judge quality; it prevents the embarrassing omissions. Lab LM12-L4 asks you to write a related checker for claims.

REQUIRED = ["problem", "data", "model", "training", "results", "limitations", "reproducibility"]

def check_report(report):
    """report: {'sections': {name: text}, 'claims': [{'text', 'evidence'}], 'numbers': {...}}. Returns a list of problems."""
    problems = []
    for name in REQUIRED:
        text = report["sections"].get(name, "")
        if len(text.split()) < 20:
            problems.append(f"section '{name}' is missing or shorter than 20 words")
    for claim in report["claims"]:
        if not claim.get("evidence"):
            problems.append(f"claim without evidence: {claim['text']}")
    for key in ("seeds", "ntest", "baseline"):
        if key not in report["numbers"]:
            problems.append(f"missing number: {key}")
    if report["numbers"].get("seeds", 0) < 3:
        problems.append("fewer than 3 seeds")
    return problems

report = {
    "sections": {"problem": "We translate short natural-language robot commands into JSON tool calls that a validator can check before anything is executed by the robot controller.",
                 "data": "Synthetic commands generated by code, 1,056 distinct inputs, held-out bay numbers 56 to 60 for a generalisation test, no personal data and no external licence involved.",
                 "model": "A two-layer character-level GPT with 111,168 parameters, learned positions, trained from scratch on CPU with AdamW and a cosine schedule over one thousand steps.",
                 "results": "Masked training reaches 0.959 exact match on the test split and 0.820 on unseen bays, with intervals reported below; a rule-based parser scores 1.0."},
    "claims": [{"text": "exact match 0.959 on test", "evidence": "table 2, n = 97, interval 0.899 to 0.984"},
               {"text": "masking is better", "evidence": ""}],
    "numbers": {"seeds": 2, "ntest": 97},
}
for p in check_report(report):
    print("-", p)
# - section 'training' is missing or shorter than 20 words
# - section 'limitations' is missing or shorter than 20 words
# - section 'reproducibility' is missing or shorter than 20 words
# - claim without evidence: masking is better
# - missing number: baseline
# - fewer than 3 seeds

The three flagged sections are exactly those that a writer forgets once the results table is done. The claim “masking is better” has no evidence string, and the number of seeds is two: both are things a reviewer would raise. Run the checker at the end of every writing session; it is cheap, and it keeps the report honest.

The README and the reproduction guide

1. Overview,One paragraph: what the project does and its main result with the interval.
2. Setup,The Python version, the environment file or install commands, the expected run time and memory.
3. Run,One command per milestone, in order, and the files each produces.
4. Expected output,The numbers the reader should see, with tolerances for different hardware.
5. Layout,Where the code, data generator, configuration and run records are.
6. Licence and data,The licence of the code and the data statement; no secrets or weights that you may not share.

Where to go next

The companion repository of the book contains full-size implementations of the architectures in this track, a pretraining recipe and fine-tuning examples; the book’s later chapters cover adapting pretrained weights. Natural continuations of the capstone are a subword tokenizer, more seeds and a larger evaluation set, a pretrained base model instead of training from scratch (Module 6), parameter-efficient fine-tuning (Module 8) and a retrieval or tool-using application (Module 11). For the physical side, the Physical AI and VLA tracks on this site treat robots that act on what language and vision models produce.

Engineering implications and limitations

Write while you work. Draft the data and training sections in milestones 1 and 2; they are easy then and painful later.

Keep a single source of numbers. A results file (JSON) written by the evaluation script, read by the report generator. No numbers typed by hand.

Review against a rubric. Use the checks of Lab LM12-L4: baseline, seeds, interval, sample size, tuning on the test set, overlapping intervals. Ask someone else to apply it.

Cite what you used. References must be real, complete and relevant. Do not cite a paper you have not read for a claim you cannot trace to it.

Do not ship weights or data you may not share. Check licences of data and pretrained models; publish the generator and the code, not material with unclear terms. Never commit credentials.

Make the limitations section the best-edited section. It is the one that tells readers how far to trust the rest.

For robots. If the system could ever drive hardware, include a short safety statement: which actions are possible, which checks exist outside the model, what was tested, and what was not. Make clear that the report is not a safety certification.

Common misconceptions

Engineering takeaways

1. A report maps claims to evidence: question, data, model, training, results with baselines and intervals, application and safety, limitations, reproducibility.
2. Match wording to evidence: “no detectable difference (9 vs 6, p = 0.61)” instead of “clearly better”.
3. Make numbers consistent by computing them from saved counts: k/n, the Wilson interval and slice sums must agree.
4. State compute (about 1.6·10¹² FLOPs per run for the default project), environment and expected outputs; provide a README with one command per milestone.
5. Write next steps as testable questions, and run a mechanical checker plus a human review before submitting.

References and further reading

Model cards and datasheets give longer checklists for documenting models and data than this lesson; adapt them to the size of your project. The numbers quoted come from the capstone notebooks and the code shown.

• Raschka, S. (2024). Build a Large Language Model (From Scratch). Manning; companion repository named in the track resources.
• Mitchell, M. et al. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
• Gebru, T. et al. (2021). Datasheets for datasets. Communications of the ACM 64(12).
• Build a Large Language Model (From Scratch), the whole book (primary book reference)
• Publisher page for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes the track. Return to the track page to review the modules, or continue with the Physical AI and VLA tracks to see language models in robots.

This is the last lesson of the track.

Related content

Nothing below is required to finish this lesson.

• Before
  LM12-T3: Evaluating and Presenting Your System

  Evaluate on held-out data, test failure cases and demonstrate a small application.

• Lab
  LM12-L4: Capstone Milestone 4: Report Review

  Check a project report against the rubric and find the missing evidence and overclaims.