This lab works on phones, but writing code is much easier on a laptop.
LLM · Build a GPT Model from Scratch · LM4-L2 · Intermediate
A GPT has dozens of layers. Multiply a signal by 0.9 in each of 48 layers and almost nothing is left; multiply by 1.1 and it is a hundred times too large. Layer normalisation and residual connections are the two inexpensive tools that keep a deep stack well behaved.
You will see the problem in a stack of random layers, write layer normalisation, and place it correctly around a residual branch.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 measures activation scale and input retention for four block designs. Stage 2 implements layer normalisation with the exact conventions of GPT-2. Stage 3 writes the pre-LN and post-LN orderings. The layers are random matrices without training and without nonlinearities, so the lab shows how scale behaves at initialisation, not how a trained model behaves. Your code is graded on hidden vectors.
Educational simplification: 12 random linear layers of width 64, no nonlinearity, no attention, no training. Real blocks contain attention and feed-forward sublayers and are trained; the scale argument carries over qualitatively. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.