LM5-T5 — Temperature and Sampling

Learning objectives

After this lesson, you should be able to:

• Compare greedy decoding and sampling and describe the failure of each.
• Compute a softmax with temperature by hand and describe its limits (T → 0, T → ∞).
• Draw a token from a distribution with multinomial sampling and with inverse-CDF sampling.
• Write the generation loop with context cropping and temperature.

The idea: choose from a distribution

Given the logits at the last position, there are two basic ways to choose.

Greedy decoding always picks the token with the largest logit. It is deterministic and often sensible for short factual answers, but over long text it falls into loops (“the robot picks up the red box. the robot picks up the red box. …”) because the most likely continuation of a repeated phrase is more of the same, and it never takes the slightly less likely route that leads to more interesting text.

Sampling draws the next token at random, each with the probability the model assigned. Text is varied and, with a well-trained model, mostly fluent. But the long tail of unlikely tokens adds up: with a vocabulary of 50,000 tokens, each individually improbable, an occasional wild choice becomes likely over a long text, and one bad token can derail the rest.

Temperature

Before the softmax the logits are divided by a number T, the temperature. For T < 1 the differences between logits are amplified: the leader gains probability and the tail fades. For T > 1 they shrink: the distribution flattens and unlikely tokens come back. At T = 1 the model’s own distribution is used. As T → 0, sampling becomes greedy decoding; as T → ∞, every token becomes equally likely and the output is random noise. The name comes from statistical physics, where a hot system explores many states and a cold one settles into the lowest.

What temperature does not do

Temperature does not make the model know more or less. It changes how closely the choice follows the model’s ranking. A high temperature gives variety at the price of errors; a low temperature gives reliability at the price of repetition. Neither makes a wrong fact right.

The loop

Generation repeats: run the model on the sequence so far, take the logits of the last position, choose a token, append it, and go again. If the sequence outgrows the context length, only the last L tokens are fed to the model.

The mathematics: softmax with temperature and sampling

For logits z₁, …, z_V and temperature T > 0:

z_i

the logit of token i from the output head, shape (V,) at one position.

T

temperature: T = 1 is the model’s distribution; T < 1 sharpens; T > 1 flattens.

p_i(T)

the probability of token i; positive, sums to 1.

The ratio of two probabilities is exp((z_i − z_j)/T): dividing by T multiplies the log-odds between any two tokens by 1/T. Halving the temperature squares the odds.

Example. Logits [2, 1, 0]. At T = 1: exp = [7.389, 2.718, 1], sum 11.107, probabilities [0.665, 0.245, 0.090]. At T = 0.5 the scaled logits are [4, 2, 0], exp = [54.6, 7.39, 1], sum 62.99: [0.867, 0.117, 0.016]. At T = 2 the scaled logits are [1, 0.5, 0], exp = [2.718, 1.649, 1], sum 5.367: [0.506, 0.307, 0.186].

Entropy. For five tokens with logits [3, 2, 1, 0, −1] the entropy of the distribution is 0.66 bits at T = 0.5, 1.44 bits at T = 1, 2.01 bits at T = 2 and 2.27 bits at T = 5; the maximum, for a uniform distribution of 5 tokens, is log₂ 5 = 2.32 bits. Entropy rises monotonically with temperature.

Sampling. Drawing a token with probability p_i can be done by inverse-CDF sampling: draw u uniformly in [0, 1), compute the running sum of the probabilities in a fixed order, and return the first token whose running sum exceeds u. PyTorch’s torch.multinomial does this efficiently. Using a seeded generator makes a sample reproducible.

Greedy decoding is argmax_i z_i, the same for any T > 0 in the limit T → 0, and needs no softmax at all.

A worked example: temperature, sampling and a generation loop

The first part computes the probabilities above and checks the sampling frequencies against them. The second part writes the generation loop with temperature and cropping, using a toy “model” that gives fixed logits so that the output is predictable.

import torch

z = torch.tensor([2.0, 1.0, 0.0])
for T in (0.5, 1.0, 2.0):
    print(T, [round(v, 3) for v in torch.softmax(z / T, dim=-1).tolist()])
# 0.5 [0.867, 0.117, 0.016]
# 1.0 [0.665, 0.245, 0.09]
# 2.0 [0.506, 0.307, 0.186]

def entropy_bits(p):
    return -(p * p.log2()).sum().item()

z5 = torch.tensor([3.0, 2.0, 1.0, 0.0, -1.0])
print([round(entropy_bits(torch.softmax(z5 / T, -1)), 2) for T in (0.5, 1, 2, 5)])   # [0.66, 1.44, 2.01, 2.27]

# sampling frequencies match the probabilities
g = torch.Generator().manual_seed(0)
draws = torch.multinomial(torch.softmax(z, -1), 10000, replacement=True, generator=g)
freq = torch.bincount(draws, minlength=3).float() / 10000
print([round(v, 1) for v in freq.tolist()])             # [0.7, 0.2, 0.1]

# inverse-CDF sampling with a given random number u
def inverse_cdf(p, u):
    total = 0.0
    for i, v in enumerate(p):
        total += v
        if u < total:
            return i
    return len(p) - 1
print([inverse_cdf([0.665, 0.245, 0.09], u) for u in (0.1, 0.7, 0.95)])   # [0, 1, 2]

# the generation loop (greedy when T is None), with context cropping
def generate(model, idx, max_new_tokens, context, temperature=None, gen=None):
    for _ in range(max_new_tokens):
        logits = model(idx[:, -context:])[:, -1, :]
        if temperature is None:
            nxt = logits.argmax(dim=-1, keepdim=True)
        else:
            nxt = torch.multinomial(torch.softmax(logits / temperature, -1), 1, generator=gen)
        idx = torch.cat([idx, nxt], dim=1)
    return idx

toy = lambda idx: torch.tensor([[[0.0, 5.0, 1.0]]]).expand(idx.shape[0], idx.shape[1], 3)   # always prefers token 1
out = generate(toy, torch.tensor([[0]]), 6, context=4)
print(out.tolist())                                      # [[0, 1, 1, 1, 1, 1, 1]]  greedy: always token 1

The last block shows the structure of every generation function: crop, run the model, take the last position’s logits, choose, append. Only the choice rule changes between greedy decoding, temperature sampling, top-k and top-p.

Engineering implications and limitations

Never pass probabilities to multinomial after dividing by zero. A temperature of exactly 0 is a division by zero. Treat T == 0 (or very small) as greedy decoding in your code.

Apply temperature to logits, not to probabilities. Dividing probabilities by T and renormalising is a different, wrong operation.

Eval mode and no_grad. Put the model in evaluation mode (dropout off) and disable gradients during generation, or text will be noisy and memory will grow.

Seed for reproducibility. A seeded torch.Generator makes a sample repeatable; log it with the settings so that bad outputs can be reproduced.

Typical settings. Factual or code answers: temperature 0 to 0.3. General chat: 0.7 to 1.0. Creative writing: 0.9 to 1.2, usually with top-p. There is no universal best value; evaluate on your task (Module 9).

Temperature and calibration. A model that is over-confident (its probabilities are higher than its accuracy) gives better samples at a slightly higher temperature. This is different from the temperature in this lesson’s loop being a free creativity knob.

Efficiency. The naive loop recomputes the whole prefix at every step. The KV cache (Module 10) stores the keys and values of past tokens so that each step costs one new position.

For robots. When a model's output drives actions, use low temperature or constrained decoding (Module 11). A sampled wrong token in a command is an action, not a typo.

Common misconceptions

Engineering takeaways

1. p_i = exp(z_i/T) / Σ exp(z_j/T): T < 1 sharpens, T > 1 flattens, T → 0 is greedy, T → ∞ is uniform.
2. For logits [2, 1, 0]: T = 0.5 gives [0.87, 0.12, 0.02]; T = 1 gives [0.67, 0.24, 0.09]; T = 2 gives [0.51, 0.31, 0.19].
3. Entropy rises monotonically with temperature (0.66, 1.44, 2.01 bits at T = 0.5, 1, 2 for five tokens).
4. Sample with torch.multinomial or inverse-CDF; seed a generator to reproduce a draw.
5. The loop is: crop to the context, run the model, take the last position’s logits, choose, append.

References and further reading

See Chapter 5 of the book (decoding strategies to control randomness: temperature scaling) and the PyTorch documentation for torch.multinomial and torch.softmax.

• Holtzman, A. et al. (2020). The Curious Case of Neural Text Degeneration. arXiv:1904.09751.
• Build a Large Language Model (From Scratch), Chapter 5: Pretraining on Unlabeled Data (primary book reference)
• Build a Large Language Model (From Scratch), Appendix D: Adding Bells and Whistles to the Training Loop (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: truncating the distribution with top-k and top-p, and deciding when generation stops.

Related content

Nothing below is required to finish this lesson.

• Before
  LM5-T4: Overfitting, Underfitting and Data Quality

  Read training and validation curves and relate them to model size, data size and data quality.

• Lab
  LM5-L2: Compare Sampling Strategies

  Implement temperature, top-k and top-p sampling and compare their diversity and quality.