LM7-L3 — Instruction-Tune a Tiny Model and Evaluate It

Objective

Implement the prompt template, the masked label sequence with an end-of-turn token, padding and collation, the masked loss, greedy generation until the end token, per-task exact-match evaluation and a contamination check, so that all seven self-checks pass, and explain why each piece is needed.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU only, about 2 GB of RAM, a few minutes. CPU only; about 2 GB of RAM; the whole notebook takes about one to two minutes (about 30 seconds of instruction tuning, 15 seconds for the memorising model, and generation for evaluation). The model has about 355,000 parameters and works on characters. Nothing is downloaded.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab instruction-tune-a-tiny-model.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

1. Template. Write build_prompt with the Instruction, Input and Response sections.
2. Masked labels. Write make_pair: ids for prompt, response and end token; labels −100 over the prompt.
3. Batching. Pad ids with a padding token and labels with −100, and shift by one for next-token training.
4. Loss. Write the cross-entropy that ignores −100, and check it against a manual computation.
5. Generate. Write greedy generation that stops at the end token; see what the base model does with an instruction.
6. Evaluate. Write exact-match evaluation per task and the contamination check.
7. Tune and compare. Instruction-tune, evaluate on held-out words, then train a memorising model on a small set and see how a leaky benchmark inflates its score.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

• Self-check 1–4: the template matches exactly; labels are −100 over the prompt and end with the end-of-turn token; padding positions are ignored; the masked loss equals the mean over non-ignored positions and ignores changes at masked positions.
• Self-check 5: the base model does not answer the instruction (it produces word-like characters instead).
• Self-check 6: the held-out words never appear in training (no overlap), the leaky benchmark contains 60 items from the small training set, and the base model scores near 0 on every task.
• Self-check 7: after 1,000 steps the tuned model reaches a mean exact-match of at least 0.6 on held-out words (in the run used to prepare this lab it was 1.0 on all three tasks). A second model tuned on only 40 words memorises them: in the prepared run it scored 0.0 on clean held-out words and 0.5 on the leaky benchmark, which is the inflation a contaminated benchmark produces. Exact numbers vary with the PyTorch version, the assertions have margins.

How to know you are done

• All seven self-checks print “passed” when the notebook is run from the top.
• You can explain what happens if the prompt is not masked, and what happens if the end token is not in the labels.
• You can say why evaluation words must differ from training words, and how contamination inflates a score.

The site cannot run or grade the notebook for you. The self-checks inside the notebook are your verification. The reference solution was executed with PyTorch on a CPU when this module was prepared. The tasks are rule-based string operations on random letters, and a 355,000-parameter character model learns them; this demonstrates the training mechanics, not the quality of instruction-tuned language models on real instructions.

If something goes wrong

• The template check fails by one character: compare with the printed expected string; the response section ends with a single newline and sections are separated by a blank line.
• Shapes are off by one after collating: inputs are ids[:, :-1] and targets labels[:, 1:]; both have length n−1.
• The tuned model never stops: check that the end token is both in ids and in labels and that the generation loop breaks on it.
• Accuracy is low after tuning: run the 1,000 steps in full; with 250 steps the model has not yet learned the rules.

Theory for this lab

• LM7-T1: Why Base Models Do Not Follow Instructions
• LM7-T5: Evaluating Instruction Following and Data Contamination

Reference material

• Build a Large Language Model (From Scratch), Chapter 7: Finetuning to Follow Instructions

Instruction Fine-Tuning labs

1. LM7-L1: Format Instruction Data with Templates
2. LM7-L2: Implement Loss Masking
3. LM7-L3: Instruction-Tune a Tiny Model and Evaluate It: this lab.

Next lab: LM8-L1: Estimate Trainable Parameters and Memory.