This lab works on phones, but writing code is much easier on a laptop.
LLM · Modern LLM Architecture and Inference · LM10-L1 · Intermediate
Generating text one token at a time would repeat enormous amounts of work if the model recomputed the whole past at every step. The key-value cache stores the past keys and values so that each step only does the new token’s work. The price is memory, which grows with the context and with the number of users.
You will fit a serving workload into a GPU, write the cache-size formula, and count the work that the cache saves.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 explores how key-value heads and cache precision decide whether long-context serving fits in memory. Stage 2 writes the cache-size formula. Stage 3 counts operations with and without a cache. Only formulas and operation counts are computed; no model is run. Your code is graded on hidden configurations.
Educational simplification: one generic 7-billion-parameter configuration in Stage 1, a simplified operation count (projections and the two attention products of one layer, ignoring the feed-forward layers, normalisation and softmax), no paging or fragmentation of the cache. All configurations are generic.
Get notified when new robotics labs are released. One email per new lab, nothing else.