LM1-T2 — Autoregressive Generation

Learning objectives

After this lesson, you should be able to:

• Describe the autoregressive generation loop and state what is fed back at each step.
• Compare greedy decoding and sampling and predict their behaviour.
• Apply temperature to a set of logits and compute the resulting probabilities.
• Explain stopping criteria and why generation is inherently sequential.

The idea: predict, choose, append, repeat

Suppose the prompt is “the robot picks up the”. The model returns a distribution over the next token. You choose a token, say “red”, and append it: “the robot picks up the red”. Now you ask the model again, with the longer text. The answer is a new distribution, because the context changed. You choose “box”, append it, and so on until the model produces an end-of-text token or you reach a length limit.

“Autoregressive” means that each output depends on the model’s own earlier outputs. The same trained network is called again and again; nothing inside it remembers the previous call, so the whole text so far must be given as input every time (Lesson LM10-T5 shows how caching avoids recomputing it).

Choosing the token: decoding rules

The model gives probabilities; a decoding rule turns them into one token.

1. Greedy decoding takes the most probable token every time. It is deterministic: the same prompt gives the same text. It is also repetitive and can miss good continuations that begin with a less likely token.
2. Sampling draws a token at random in proportion to its probability. A token with 25% probability is chosen about a quarter of the time. The text varies from run to run and is more diverse, and also sometimes strays.
3. Temperature, top-k and top-p adjust sampling to trade diversity against reliability. Module 5 treats them in detail; the lab here uses temperature.

Why the prompt matters

The prompt is simply the first part of the text. The model has no other way to know what you want: instructions, examples and background all enter as context tokens. A base model, trained only to continue text, continues your prompt as if it were a document. Making a model answer questions instead of continuing them is the subject of Module 7.

Stopping

The loop needs a rule to stop. Usually a special end-of-text token is part of the vocabulary and the model learns to predict it when a document is complete; generation stops when it is chosen. A maximum number of new tokens is always set as well, as a safety limit.

The mathematics: logits, softmax and temperature

A neural model does not output probabilities directly. For a context it outputs one real number per vocabulary token, the logits z₁, …, z_V. The softmax turns them into probabilities, and a temperature T > 0 rescales them first:

z_i

the logit of token i: higher means the model prefers that token. Logits can be any real number, positive or negative.

T

the temperature. T = 1 is the model’s own distribution; T < 1 sharpens it (the most likely token gains); T > 1 flattens it; T → 0 approaches greedy decoding.

p_i

the probability of token i; all p_i are positive and add up to 1.

V

the vocabulary size; j runs over all tokens in the sum.

Dividing the logits by T does not change their order, so the most likely token stays most likely; only the gaps change. Sampling from the result, using a uniform random number u in [0, 1), means walking through the tokens in a fixed order, adding probabilities, and taking the first token at which the running total exceeds u (the inverse-CDF method of Lab LM1-L1).

Cost of generation. Generating m new tokens needs m sequential model calls, because token t + 1 depends on token t. The calls cannot be done in parallel for one text, which is why long outputs take time proportional to their length. Training does not have this limitation (Module 5): all positions of a known text are predicted at once.

Probability of the generated text. The probability of a generated sequence is the product of the probabilities of the chosen tokens. Greedy decoding does not maximise this product over whole sequences (a locally best token may lead to a poor continuation); finding the globally best sequence needs search methods such as beam search, which the book does not require.

A worked example: temperature on three logits

After a prompt, a model gives logits z = (2.0, 1.0, 0.0) for the tokens “red”, “green” and “blue”. At T = 1:

At T = 0.5 the logits become (4, 2, 0): exp = (54.60, 7.389, 1.000), sum 62.99, so p = (0.867, 0.117, 0.016). “red” is now chosen 87% of the time instead of 67%. At T = 2 the logits become (1, 0.5, 0): exp = (2.718, 1.649, 1.000), sum 5.367, so p = (0.506, 0.307, 0.186): the three tokens are much closer.

Greedy decoding picks “red” at every temperature. Sampling at T = 2 picks “blue” about one time in five.

The loop in Python

import math, random

def softmax(logits, T=1.0):
    m = max(logits)
    exps = [math.exp((z - m) / T) for z in logits]   # subtract the max: avoids overflow, same result
    s = sum(exps)
    return [e / s for e in exps]

def generate(next_logits, tokens, vocab, max_new=20, T=1.0, end="<end>", seed=0):
    rng = random.Random(seed)
    for _ in range(max_new):
        probs = softmax(next_logits(tokens), T)        # the model: tokens -> logits
        tok = rng.choices(vocab, weights=probs)[0]     # sampling; use vocab[probs.index(max(probs))] for greedy
        tokens = tokens + [tok]
        if tok == end:
            break
    return tokens

The model is the function next_logits. Subtracting the largest logit before exponentiating prevents overflow and does not change the result, because the common factor cancels in the ratio. The seed makes the sampling reproducible, which matters when you evaluate or debug a model (Module 9).

Engineering implications and limitations

Decoding settings are part of the system. The same model can look careful or wild depending on temperature. Report them with any result, and fix the seed when comparing.

Use sampling or greedy deliberately. For tasks with one right answer (extraction, classification, code under test) low temperature or greedy decoding is usually better; for creative variety, sampling. Neither makes a false statement true.

Always set a maximum length. A model can fail to emit an end token and run on, repeat itself, or loop. A hard limit protects latency and cost.

Expect latency to grow with output length. Because tokens are generated one after another, a 500-token answer takes about 10 times as long as a 50-token one, even though the prompt processing is parallel. Module 9 and Module 10 quantify this.

Do not feed raw model text into actions. A sampled sequence can differ from run to run. In a robot, text from a model must pass through parsing, validation and a safety layer before anything moves (Module 11); the Physical AI track shows one such design.

Autoregression compounds errors. Once a wrong token is appended, it becomes part of the context and influences everything after it. This is one reason long outputs drift.

Common misconceptions

Engineering takeaways

1. Generation is a loop: predict a distribution, choose a token, append it, repeat until an end token or a length limit.
2. Greedy decoding is deterministic and repetitive; sampling follows the probabilities and varies from run to run.
3. Softmax with temperature: p_i = exp(z_i/T) / Σ exp(z_j/T); T below 1 sharpens, T above 1 flattens, the order of tokens never changes.
4. Output tokens are generated sequentially, so latency grows with output length.
5. Fix the seed and the decoding settings whenever results are compared.

References and further reading

Chapter 1 and the text-generation parts of Chapter 5 of the book are the core path; the paper is optional.

• Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR: why greedy and pure sampling both have problems.
• Build a Large Language Model (From Scratch), Chapter 1: Understanding Large Language Models (primary book reference)
• Companion site for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what happens when the model is built (training) and when it is used (inference), and where pretraining and fine-tuning fit.

Related content

Nothing below is required to finish this lesson.

• Before
  LM1-T1: What a Language Model Is

  Define a language model as a probability distribution over the next token and compute probabilities from counts.

• Lab
  LM1-L1: Explore Next-Token Probabilities and Decoding

  Build a count-based next-token model and compare greedy decoding with sampling.

• Lab
  LM1-L2: Trace Text to Generated Output

  Follow a prompt through tokens, context, probabilities and a generated continuation, and see how context changes predictions.