This lab works on phones, but writing code is much easier on a laptop.
LLM · Embeddings and the Mathematics of Attention · LM3-L2 · Intermediate
A language model is trained to predict each token from the tokens before it. If attention could see the future, it would simply copy the answer and learn nothing useful.
You will test for leakage, write the mask so that rows stay normalised, and handle padding the way batches require.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 measures information leaking from a future token into earlier outputs under three masking methods. Stage 2 writes the causal softmax. Stage 3 combines it with left padding. The matrices are random numbers and the lab teaches the mask, not a trained model. Your code is graded on hidden matrices.
Educational simplification: 5 by 5 matrices, a single head, plain Python lists, random scores. Real models apply the same mask to a (batch, heads, tokens, tokens) tensor and usually use a large negative number rather than −∞ for numerical convenience. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.