LM12-L3 — Capstone Milestone 3: Evaluation and Application

Objective

Implement an error taxonomy, an exact paired sign test, a schema-and-policy validator for model output and an application wrapper, so that all four self-checks pass, and report test and out-of-distribution results with intervals, the paired comparison, and the application-level outcome including what the policy refuses.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU only, about 2 GB of RAM, a few minutes. CPU only; about 2 GB of RAM. Two models are trained again (about 25 seconds each), then evaluated; the notebook takes roughly one to two minutes.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab capstone-milestone-3-evaluation-and-application.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

1. Error taxonomy. Classify every prediction as correct, invalid JSON, wrong tool or wrong arguments.
2. Evaluate. Run both models on the test and out-of-distribution splits; report Wilson intervals and the taxonomy.
3. Paired test. Compare the models example by example with an exact two-sided sign test.
4. Safety layer. Validate keys, types, ranges and a speed limit in a fixed order; wrap generation in an application.
5. Property test. Run hostile and ordinary inputs and check that every accepted call satisfies the policy.
6. Application-level result. Report executed, refused and correct-but-refused counts, and write what the report must say.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

• Self-check 1: the taxonomy partitions hand-made predictions into correct, invalid JSON, wrong tool and wrong arguments.
• Self-check 2: for the masked model (reference run) 93 of 97 test examples and 73 of 89 held-out examples are exactly right, with Wilson intervals (0.899, 0.984) and (0.728, 0.886).
• Self-check 3: the sign test gives p = 0.1094 for 8 against 2 and p = 0.002 for 10 against 0; in the reference run masked and unmasked disagree on 15 held-out examples (9 against 6, p = 0.6072), which is no evidence of a difference.
• Self-check 4: the validator refuses each class of bad call with the stated reason, and no call accepted by the application violates the policy; the application executes 72 of 97 test commands, 71 of them exactly right, and refuses 25, of which 22 are correct commands that are too fast.

How to know you are done

• All four self-checks print “passed” when the notebook is run from the top.
• You can explain why the paired test finds no difference where the pooled intervals of milestone 2 seemed separated.
• You can state what the property test proves (accepted calls satisfy the policy) and what it does not (that the model is right, or that the policy is complete).

The site cannot run or grade the notebook for you. The self-checks inside it are your verification. The reference solution was executed with PyTorch on a CPU when this module was prepared; the quoted numbers come from that run and may differ slightly on your machine. The application executes nothing real: it only decides whether a call would be accepted.

If something goes wrong

• The taxonomy counts do not add up: every prediction must fall into exactly one class; check the order: equality first, then parsing, then tool, then arguments.
• A boolean passes as a bay number: in Python True is an int; test for bool explicitly before accepting integers or numbers.
• The property test fails: an accepted call must have passed every rule; check the order of the range and speed checks.
• No PyTorch installed: run pip install torch in your virtual environment; the CPU build is enough.

Theory for this lab

• LM12-T3: Evaluating and Presenting Your System

Reference material

• Build a Large Language Model (From Scratch), the whole book
• Publisher page for the book

Capstone: Build and Evaluate Your Own LLM System labs

1. LM12-L1: Capstone Milestone 1: Dataset, Tokenizer and Pipeline
2. LM12-L2: Capstone Milestone 2: Model and Training Experiment
3. LM12-L3: Capstone Milestone 3: Evaluation and Application: this lab.
4. LM12-L4: Capstone Milestone 4: Report Review

Next lab: LM12-L4: Capstone Milestone 4: Report Review.