This lab works on phones, but writing code is much easier on a laptop.
LLM · Text Data and Tokenization · LM2-L2 · Intermediate
A tokenizer with one token per word fails on every new word. One token per character makes every sentence long. Byte-pair encoding finds the middle: it builds frequent pieces out of characters, and unseen words fall back to the pieces.
You will learn the merge rules from a tiny corpus and use them to encode words the corpus never contained.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 shows the trade-off between vocabulary size and sequence length. Stage 2 implements the merge-learning algorithm. Stage 3 encodes words by replaying the merges in order. The corpus is a handful of invented word counts, and BPE works on characters with an end-of-word marker; it is a faithful miniature of the algorithm, not a production tokenizer. Your code is graded on hidden corpora and words.
Educational simplification: a handful of invented words, characters instead of bytes, an explicit end-of-word marker, and deterministic tie-breaking. Production tokenizers (such as the one used by GPT-2) operate on bytes of very large corpora, pre-split text with rules, and have tens of thousands of merges. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.