LM12-L2 — Capstone Milestone 2: Model and Training Experiment

Objective

Implement majority and nearest-neighbour baselines, a seeded AdamW loop with warm-up and cosine decay, a run record, a Wilson interval and a grouped summary so that all four self-checks pass, and report the masked-versus-unmasked comparison with its limits.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU for the small configuration; GPU optional. CPU only; about 2 GB of RAM. Each of the four training runs (1,000 steps, a 2-layer model of width 64) takes roughly 20 to 30 seconds on a laptop; the notebook needs about two minutes in all.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab capstone-milestone-2-model-and-experiment.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

1. Baselines. Majority target and nearest neighbour by character bigrams; compare with the given rule-based parser.
2. Training loop. AdamW, warm-up, cosine decay, seeded batches; check the loss falls and that identical seeds give identical curves.
3. Records and summary. Write a run record, a Wilson interval and a grouped summary; test them on made-up numbers.
4. Experiment. Run masked and unmasked training with two seeds each and log all four runs.
5. Interpret. Compare the intervals and the spread between seeds; write what you can and cannot claim.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

• Self-check 1: the majority and nearest-neighbour baselines score below 0.05 and 0.2; the rule-based parser scores 1.0 on all splits of this synthetic task.
• Self-check 2: two runs with the same seed have identical loss lists on CPU; different seeds differ; the loss falls in 40 steps.
• Self-check 3: records, Wilson intervals (18 of 20 gives (0.699, 0.972)) and summaries match hand-computed values.
• Self-check 4: in the reference run all four models reach at least 0.9 exact match on validation; masked training gives out-of-distribution exact match of 0.820 and 0.843 and unmasked training 0.787 and 0.551 for the two seeds, with pooled Wilson intervals (0.77, 0.879) and (0.596, 0.733).

How to know you are done

• All four self-checks print “passed” when the notebook is run from the top.
• You can explain why two seeds and one run per seed cannot establish an effect even when the pooled intervals are separated.
• You can state what the rule-based baseline implies for the value of the learned model in this task.

The site cannot run or grade the notebook for you. The self-checks inside it are your verification. The reference solution was executed with PyTorch on a CPU when this module was prepared; the numbers quoted come from that run and may differ slightly on your machine. The task is synthetic and a rule-based parser solves it exactly.

If something goes wrong

• Two runs with the same seed differ: create the batch generator once, before the loop, and call torch.manual_seed(seed) before building the model.
• The loss does not fall: check that inputs are X[:, :-1] and targets Y[:, 1:], and that padding labels are −100.
• Numbers differ slightly from those quoted: floating-point results can depend on the PyTorch version and CPU; the assertions leave a margin, and your own numbers are the ones to report.
• No PyTorch installed: run pip install torch in your virtual environment; the CPU build is enough.

Theory for this lab

• LM12-T2: Experiment Design and Reproducibility

Reference material

• Build a Large Language Model (From Scratch), the whole book
• Publisher page for the book

Capstone: Build and Evaluate Your Own LLM System labs

1. LM12-L1: Capstone Milestone 1: Dataset, Tokenizer and Pipeline
2. LM12-L2: Capstone Milestone 2: Model and Training Experiment: this lab.
3. LM12-L3: Capstone Milestone 3: Evaluation and Application
4. LM12-L4: Capstone Milestone 4: Report Review

Next lab: LM12-L3: Capstone Milestone 3: Evaluation and Application.