LM9-T1 — Perplexity and Its Limits

Learning objectives

After this lesson, you should be able to:

• Compute perplexity from the mean token loss and interpret it.
• Convert to bits per byte to compare models with different tokenizers.
• List the properties of a model that perplexity does not measure.
• Report a perplexity comparison with the text, tokenizer and an uncertainty.

The idea: how surprised is the model by real text?

A language model assigns probabilities to the next token. On held-out text, the average negative log-probability of the true tokens is the cross-entropy; its exponential is the perplexity (Lesson LM5-T1). A perplexity of 18 means that on average the model is as uncertain as if it were choosing uniformly among 18 tokens. Lower is better, and 1 would mean perfect prediction.

What makes it useful

Perplexity is computed on any text, with no labels. It correlates with downstream quality within a model family trained on similar data. It decreases smoothly during training, so it can detect divergence and overfitting. It is the metric behind scaling laws (Lesson LM5-T4).

What it depends on

• The text. Perplexity on code, legal text and chat transcripts differ by large factors for the same model. A number is only meaningful with the data set named.
• The tokenizer. Perplexity is per token. A tokenizer that splits text into more, smaller tokens makes each token easier to predict and the perplexity lower, without the model knowing more. Two models with different tokenizers cannot be compared by perplexity directly.
• The context. Perplexity computed with a long context is lower than with a short one; state how the text was windowed.

What it does not measure

• Usefulness. A model can predict web text well and still not follow instructions (Lesson LM7-T1). Instruction-tuned models often have worse perplexity on web text than their base models, while being far more useful.
• Truthfulness. Fluent, probable text can be false.
• Generation quality. Greedy and low-temperature generation can loop even from a model with excellent perplexity.
• Calibration and safety. Perplexity says nothing about whether the model refuses what it should, or knows what it does not know.
• Distribution shift. Good perplexity on the benchmark text does not guarantee good behaviour on your data.

Contamination

If the evaluation text appeared in the training data, the model has partly memorised it and the perplexity is too low. Check for overlap (Lesson LM7-T5).

The mathematics: perplexity, bits per byte and uncertainty

For a text of N tokens with model probabilities p_i for the true tokens:

N

number of tokens scored (not counting the first token if there is no context for it).

p_i

probability the model gave to the true token i given the preceding context.

H

mean negative log-likelihood in nats per token.

To compare across tokenizers, convert to bits per byte using the number of bytes B in the text:

where N/B is the number of tokens per byte (the inverse of bytes per token). Lower is better.

Example. Model A: H = 2.9 nats per token, 4 bytes per token (N/B = 0.25): PPL = e^2.9 = 18.2, BPB = 2.9 × 0.25 / 0.693 = 1.046. Model B: H = 2.0 nats per token, 2.5 bytes per token (0.4 tokens per byte): PPL = e^2.0 = 7.4, BPB = 2.0 × 0.4 / 0.693 = 1.154. B has the lower perplexity and the higher (worse) bits per byte: A predicts the text better.

Uncertainty. The per-token losses are not independent (neighbouring tokens are correlated), but a rough standard error of H is the standard deviation of per-token losses divided by √N, and for a comparison of two models on the same text the standard error of the mean per-token difference (Lab LM9-L1 Stage 3). For N = 10,000 tokens and a per-token standard deviation of 2.5 nats, SE = 0.025 nats, so PPL is known to about ±2.5% (1.96 × 0.025 = 0.05 in the exponent). Differences smaller than that are noise; better, use document-level resampling.

Perplexity and compression. BPB is the length, in bits per byte, of the text if encoded by an arithmetic coder driven by the model, so a model with BPB 1.05 could compress text to 13% of its size (1.05/8). This connects language modelling and compression: a better predictor is a better compressor.

A worked example: perplexity, bits per byte and a fair comparison

The code computes the numbers above, a perplexity from a list of token probabilities, and an approximate interval for a mean loss.

import math

def perplexity(probs):
    h = -sum(math.log(p) for p in probs) / len(probs)
    return h, math.exp(h)

h, ppl = perplexity([0.5, 0.1, 0.8, 0.25])
print(round(h, 3), round(ppl, 3))                 # 1.151 3.162

def bits_per_byte(nats_per_token, bytes_per_token):
    return nats_per_token / bytes_per_token / math.log(2)

a = bits_per_byte(2.9, 4.0)
b = bits_per_byte(2.0, 2.5)
print(round(math.exp(2.9), 1), round(math.exp(2.0), 1), round(a, 3), round(b, 3))   # 18.2 7.4 1.046 1.154
print("better by bits per byte:", "A" if a < b else "B")                              # A

# compression view: size of the text if coded with the model, relative to raw bytes
print(round(a / 8, 3))                                                                  # 0.131

# an approximate standard error of the mean loss
N, sd = 10_000, 2.5
se = sd / math.sqrt(N)
print(round(se, 3), f"perplexity range {math.exp(2.9 - 1.96 * se):.1f} to {math.exp(2.9 + 1.96 * se):.1f}")   # 0.025 perplexity range 17.3 to 19.1

The interval 17.3 to 19.1 is about ±5% in perplexity for a 0.05-nat uncertainty in the loss. Real per-token losses are correlated, so the true uncertainty is larger than this naive estimate; resampling whole documents gives a more honest number.

A short reading guide

Engineering implications and limitations

Always state the data and the tokenizer. A perplexity without them is not a result.

Use bits per byte across tokenizers. Count the bytes of the evaluated text in UTF-8, and the tokens scored.

Mind the windowing. Perplexity over fixed windows of length L penalises the first tokens of each window. A sliding window with overlap and scoring only the new tokens gives a lower, fairer figure; state which you used.

Keep a general-text perplexity next to the task metrics. If fine-tuning improves your task and general perplexity rises sharply, you are forgetting (Lesson LM6-T1).

Use perplexity for monitoring, not for decisions about quality. It is an excellent training signal and a poor stand-in for an application metric.

Check for degenerate low values. Evaluation text full of repetitions or boilerplate gives very low perplexity regardless of model quality; deduplicate the evaluation set.

Use document-level resampling for intervals. Tokens within a document are correlated; resample documents (bootstrap) rather than tokens (Lesson LM9-T3).

Common misconceptions

Engineering takeaways

1. PPL = exp(mean token loss in nats); it is the effective number of choices per token.
2. Perplexity depends on the text, the tokenizer and the windowing; compare across tokenizers with bits per byte, BPB = (H/ln 2)·(tokens per byte).
3. Example: PPL 18.2 with 4 bytes/token is better (1.05 BPB) than PPL 7.4 with 2.5 bytes/token (1.15 BPB).
4. Perplexity does not measure instruction following, truthfulness, generation quality, calibration or safety.
5. Report it with the data, tokenizer, context and an uncertainty from resampling documents.

References and further reading

See Chapter 5 of the book (calculating the text generation loss, and training and validation losses). The bits-per-byte conversion and the worked numbers in this lesson are standard and can be reproduced with the code shown.

• Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: task evaluation and test sets, which measure what the application needs.

Related content

Nothing below is required to finish this lesson.

• Before
  LM8-T5: Choosing a Base Model and Managing Checkpoints

  Select a base model for a task and hardware budget and keep experiments reproducible.

• Lab
  LM9-L1: Compare Two Checkpoints

  Compare two models with perplexity and a paired bootstrap, and decide whether the difference is real.