This lab works on phones, but writing code is much easier on a laptop.
LLM · Build a GPT Model from Scratch · LM4-L1 · Beginner
Before building a model it helps to know how big it will be and what its tensors look like. A parameter count is a quick check that your code matches the paper, and a shape trace is the quickest way to find where a bug lives.
You will design a model of a given size, write the counter, and trace shapes from token embeddings to logits.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 shows how parameters divide among embeddings, attention, feed-forward layers and normalisation. Stage 2 writes the exact counter. Stage 3 traces the shapes of the main tensors. Only the arithmetic of a configuration is computed; no neural network is built, trained or run here. The PyTorch notebook of this module builds the model for real. Your code is graded on hidden configurations.
Educational simplification: a fixed vocabulary and context length in Stage 1, a feed-forward expansion factor of 2 or 4, no weight sharing other than the output head, and only five tensors in the shape trace. All configurations in this lab are invented except the GPT-2 small reference case.
Get notified when new robotics labs are released. One email per new lab, nothing else.