LM9-L3 — Evaluate a Tiny GPT End to End

Objective

Implement exact-match scoring, seeded greedy and temperature decoding, a harness that returns per-item results, a confidence interval, a paired sign test and inference cost measurements, so that all six self-checks pass, and explain what the intervals, the p-value and the seed spread allow you to conclude.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU only, about 2 GB of RAM, a few minutes. CPU only; about 2 GB of RAM; the notebook takes about two to four minutes (tuning two checkpoints, then several evaluations of 60 items, each generating up to 24 characters per item). Latency numbers depend on your machine and thread count.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab evaluation-harness-for-a-tiny-gpt.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

1. Scoring. Write the normalised exact-match function.
2. Decoding. Write generation with greedy and seeded temperature sampling; check determinism and seed behaviour.
3. Harness and interval. Return one 0/1 result per item and a 95% interval for the accuracy.
4. Paired test. Count the items where the checkpoints disagree and compute the exact sign-test p-value.
5. Cost. Measure time to first token and tokens per second; compute parameter and KV-cache bytes.
6. Experiments. Compare the two checkpoints, repeat the sampled evaluation over five seeds, and read the table.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

• Self-check 1–2: scoring is exact after stripping whitespace; greedy decoding and a seeded sample are reproducible; different seeds give different samples for an imperfect model.
• Self-check 3–4: the interval for 45 of 50 is 0.90 [0.82, 0.98]; the sign test gives p = 0.0352 for a 12-to-3 split of disagreements.
• Self-check 5: the KV cache of the model for 100 tokens is 2·2·64·100·4 = 102,400 bytes; latency numbers are positive and machine dependent.
• Self-check 6: the long checkpoint scores higher than the short one, at least 0.6, and the paired test finds the difference significant (p below 0.05); sampling at temperature 1.0 varies from seed to seed. In the run used to prepare this lab: short 0.47 [0.34, 0.59], long 1.00 (interval [1.00, 1.00], where the normal approximation fails), 32 items only the long checkpoint got right and none the other way (p below 10⁻⁹); sampled accuracy over five seeds 0.93 to 1.00 (mean 0.97). Exact accuracies depend on the PyTorch version.

How to know you are done

• All six self-checks print “passed” when the notebook is run from the top.
• You can explain why the paired test is more sensitive than comparing two intervals.
• You can say what the harness cannot tell you about the model.

The site cannot run or grade the notebook for you. The self-checks inside the notebook are your verification. The reference solution was executed with PyTorch on a CPU when this module was prepared; accuracies, p-values and latency will differ slightly on your machine. Latency is measured for a single request on a CPU and says little about serving at scale.

If something goes wrong

• The paired test is not significant: check that both checkpoints are evaluated on the same items in the same order, and that the 0/1 lists have the same length.
• Sampled accuracy is identical for all seeds: make sure the seed is passed to the torch.Generator and the temperature is not None.
• Latency is strange on the first call: the first forward pass includes one-off initialisation; call the function twice and use the second result, or average several runs.
• The notebook is slow: reduce the number of seeds from five to three; the assertions do not depend on it.

Theory for this lab

• LM9-T5: Inference Latency and Memory
• LM9-T6: Responsible Evaluation and Model Limitations

Reference material

• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models

Evaluation, Inference and Reliability labs

1. LM9-L1: Compare Two Checkpoints
2. LM9-L2: Build an Evaluation Harness
3. LM9-L3: Evaluate a Tiny GPT End to End: this lab.

Next lab: LM10-L1: KV-Cache Memory and Behaviour.