This lab works on phones, but writing code is much easier on a laptop.
LLM · Pretraining and Text Generation · LM5-L1 · Beginner
Training a language model means making one number small: the average surprise of the model at the token that actually came next. That number is the cross-entropy, and its exponential is the perplexity.
You will see how it changes with a better model, write the formula, and make it numerically safe.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 compares count-based models of different context length on a training and a validation text. Stage 2 computes loss and perplexity from probabilities. Stage 3 computes the loss from logits with the log-sum-exp trick. The models are count-based and the text is a few hundred invented tokens; nothing is trained by gradient descent here. Your code is graded on hidden probability tables and logits.
Educational simplification: an add-α n-gram model on invented robot sentences, natural logarithms throughout, no batching. A neural language model produces logits instead of counts, and its loss is computed in exactly the way of Stages 2 and 3. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.