This lab works on phones, but writing code is much easier on a laptop.
LLM · Text Data and Tokenization · LM2-L1 · Beginner
A model cannot read “the robot picks up the box”. It reads a list of integers. Every difference between “The” and “the”, or between “box” and “box.”, is a different integer unless the tokenizer decides otherwise.
You will make those decisions, build the vocabulary, and write the two functions that go between text and numbers.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 explores normalisation, punctuation and frequency cut-offs. Stage 2 builds a vocabulary with special tokens in a deterministic order. Stage 3 writes encode and decode. The tokenizer is word-level and the text is invented; it is a teaching simulation, not the byte-pair tokenizer of a real GPT (see the next lab). Your code is graded on hidden inputs.
Educational simplification: a word-level tokenizer, lowercase text, a short list of punctuation marks and a few dozen invented words. Real LLM tokenizers work on subwords or bytes, handle any Unicode text, and have tens of thousands of entries. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.