Skip to the lab
FixTheRobot Explore Robotics / LLM / LM12 / LM12-L1

LLM · Notebook lab · LM12 · Advanced

LM12-L1 — Capstone Milestone 1: Dataset, Tokenizer and Pipeline

Build the data side of your own LLM project, with tests that catch leakage and masking mistakes before they cost you a training run.

  • Module 12: Capstone: Build and Evaluate Your Own LLM System
  • PyTorch notebook
  • Hardware: CPU only, about 2 GB of RAM, a few minutes
  • About 45 to 90 minutes

Most failed LLM projects fail in the data and the pipeline, not in the model. This first milestone makes the data side explicit and testable: where the examples come from, which part is held out, which characters the tokenizer knows, which positions the loss sees.

The default project is a tiny robot-command parser (natural-language command to JSON tool call) whose data are generated by code, so there are no licence or privacy questions. You may replace the generator with your own data and keep every check.

Objective

Implement a leakage-free split with a held-out out-of-distribution slice, a training-only character tokenizer with special tokens, an encoder that masks the prompt in the labels, padded batches, and a dataset report, so that all four self-checks pass and the dataset card questions can be answered from your own output.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU only, about 2 GB of RAM, a few minutes. CPU only; about 2 GB of RAM; everything runs in a few seconds after PyTorch has loaded. No downloads.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab capstone-milestone-1-data-and-pipeline.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

  1. Generate. Read the generator: 1,000 distinct move commands plus 56 commands for three rare tools.
  2. Split. Hold out unseen bay numbers, shuffle with a seed and cut 80/10/10; test for leakage and determinism.
  3. Tokenizer. Build the vocabulary from the training split only, with PAD, END and UNK; encode with a prompt mask.
  4. Batches. Pad ids with 0 and labels with −100; check the number of supervised tokens.
  5. Report. Count examples per split and tool and the sequence-length percentiles; write the dataset card answers.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

  • Self-check 1: four splits whose sizes add up to 1,056; the reference run gives 773 / 97 / 97 train, validation, test and 89 held-out examples; no shared input text; deterministic for a fixed seed.
  • Self-check 2: a vocabulary of 49 symbols (3 special tokens and 46 characters in the reference run) that covers every character of the validation and test splits; the prompt is masked in the labels; unseen characters map to UNK.
  • Self-check 3: batches of shape (16, 75) with exactly len(target) + 1 supervised tokens per row.
  • Self-check 4: the report lists 725 move_to examples against 14 to 19 for each rare tool in the training split, and the median and 95th percentile sequence lengths 64 and 72.

How to know you are done

  • All four self-checks print “passed” when the notebook is run from the top.
  • You can state, from your own output, which part of the data measures generalisation and which measures only memorisation.
  • You can name two ways in which information from the test split could leak into training in a project of your own.

The site cannot run or grade the notebook for you. The self-checks inside it are your verification. The data are synthetic and very regular; real data are messier, and a generator you wrote yourself can never reveal the phrasings you did not think of.

If something goes wrong

  • Leakage assertion fails: shuffle only the examples that are not held out, and cut the shuffled list; do not shuffle the held-out slice back in.
  • Vocabulary size differs: build it from text + "\n=>" + target of the training split only; including the other splits hides unseen-character problems.
  • Supervised token count is wrong: the prompt is text + "\n=>", which includes the three characters of the separator; the target characters and END are supervised.
  • No PyTorch installed: run pip install torch in your virtual environment; the CPU build is enough.

Theory for this lab

Reference material

Capstone: Build and Evaluate Your Own LLM System labs

  1. LM12-L1: Capstone Milestone 1: Dataset, Tokenizer and Pipeline: this lab.
  2. LM12-L2: Capstone Milestone 2: Model and Training Experiment
  3. LM12-L3: Capstone Milestone 3: Evaluation and Application
  4. LM12-L4: Capstone Milestone 4: Report Review

Next lab: LM12-L2: Capstone Milestone 2: Model and Training Experiment.