FixTheRobot Explore Robotics / LLM / LM1 / LM1-L2

This lab works on phones, but writing code is much easier on a laptop.

LLM · Understanding Large Language Models · LM1-L2 · Beginner

Trace Text to Generated Output

A chatbot answers perfectly for ten messages and then seems to forget your name. Nothing broke: your name slipped out of its context window.

You will follow a prompt through a tiny language model and see exactly how context decides the next word.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().

    Hints

    What this lab teaches

    Stage 1 shows how much context a prediction needs. Stage 2 predicts the next word from a context, backing off when the context is unseen. Stage 3 writes the generation loop with a context window and a stopping rule. The model is an n-gram count table, a simulation of the idea, not a neural network; nothing is trained. Your code is graded on hidden corpora.

    Theory for this lab

    Understanding Large Language Models labs

    1. LM1-L1: Explore Next-Token Probabilities and Decoding
    2. LM1-L2: Trace Text to Generated Output: this lab.

    Educational simplification: words instead of subword tokens, a corpus of a few dozen words, count-based prediction with backoff instead of a neural network, and greedy decoding only. A real GPT feeds token IDs through learned layers and produces a distribution over a vocabulary of tens of thousands of tokens. All data in this lab are invented.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.