LM5-T4 — Overfitting, Underfitting and Data Quality

Learning objectives

After this lesson, you should be able to:

• Recognise healthy, overfitting, underfitting and diverging curves and name the usual fix for each.
• Explain memorisation, duplicated data and train/validation leakage.
• State the power-law form of scaling laws and the compute-optimal rule of thumb of about 20 tokens per parameter.
• Estimate training tokens and compute for a given model size using C ≈ 6ND.

The idea: two curves, four stories

The training loss measures how well the model fits what it has seen; the validation loss measures how well it generalises. The way the two move relative to each other tells the story.

Memorisation

A model with enough parameters can store its training text instead of learning rules from it. On the training set it then looks excellent. On new text it fails, because nothing general was learned. In the notebook, the memorisation run (a bigger model, 600 characters of training text, 400 steps) reaches a training loss of 0.07 while the validation loss, which was 0.36 at step 100, rises to 0.62. Large language models also memorise: they can reproduce rare passages, especially ones that appear many times in the training data.

Data quality

The data set sets the ceiling. Duplicated documents make memorisation more likely and give those documents too much weight. Low-quality text (boilerplate, spam, machine-generated text) teaches the model to produce it. The same text appearing in training and validation (leakage) makes the validation loss look better than it is, and when benchmark questions leak into the training data (contamination) evaluation scores become meaningless (Module 9). Careful pipelines remove exact and near-duplicate text, filter by language and quality, and keep the validation and test sets strictly separate from training.

Scaling laws

Empirically, the loss of a language model on held-out text falls as a smooth power law when you increase the number of parameters, the amount of training data or the compute, provided the other factors are not a bottleneck. Kaplan et al. (2020) reported exponents of about 0.076 for model size and 0.095 for data size. Later work (Hoffmann et al., 2022) found that, for a fixed compute budget, earlier models were too large for the amount of data they saw, and suggested that parameters and tokens should grow together, with roughly 20 training tokens per parameter as a rule of thumb. These are empirical fits to specific model families and data; they guide planning but are not laws of nature.

The mathematics: power laws and the compute budget

A scaling law has the form

N, D

number of (non-embedding) parameters and number of training tokens.

N_c, D_c

fitted constants that depend on tokenizer, data and architecture.

α_N, α_D

fitted exponents: about 0.076 and 0.095 in Kaplan et al. (2020). Small numbers: loss falls slowly but steadily.

With α = 0.076, multiplying the model size by 10 multiplies the loss by 10^−0.076 = 0.84, a 16% reduction; doubling the size reduces it by about 5%. With α = 0.095, multiplying the data by 10 multiplies the loss by 0.80. Improvements are real but expensive, which is why the planning arithmetic matters.

Compute. Training a Transformer with N parameters on D tokens costs about

(about 2N per token for the forward pass and 4N for the backward pass; Lesson LM4-T5). With the rule of thumb D ≈ 20N, a compute-optimal run has C ≈ 120 N². For GPT-2 small (N = 124 million): D ≈ 2.5 billion tokens and C ≈ 6 · 1.24·10⁸ · 2.5·10⁹ ≈ 1.9·10¹⁸ FLOPs. A GPU that sustains 10¹⁴ FLOP/s would need about 5 hours for this; a CPU at 5·10¹⁰ FLOP/s about 14 months. The notebook model (107,000 parameters, 819,200 tokens) needs 6 · 1.07·10⁵ · 8.2·10⁵ ≈ 5.2·10¹¹ FLOPs: seconds.

The generalisation gap. Define gap = validation loss − training loss. A healthy run has a small gap that grows slowly. A gap that grows while validation loss rises is overfitting; a large gap from the start suggests that the validation data are different from the training data.

A worked example: diagnosing four runs

The table summarises three runs of the notebook LM5-L3 and one illustrative case (loss in nats; initial loss ln 22 = 3.09).

The rules used by diagnose in the notebook follow this reading: divergence is a loss above twice its initial value; underfitting is a final training loss above half of the initial one; overfitting is a gap of at least 0.25 with validation above its own minimum.

import math

def diagnose(history):
    """history: list of (step, train_loss, val_loss)."""
    tr0 = history[0][1]
    if any(not math.isfinite(a) or not math.isfinite(b) for _, a, b in history) or max(a for _, a, _ in history) > 2 * tr0:
        return "diverged"
    tr, va = history[-1][1], history[-1][2]
    if tr > 0.5 * tr0:
        return "underfitting"
    if va - tr >= 0.25 and va >= min(v for _, _, v in history) + 0.05:
        return "overfitting"
    return "healthy"

healthy = [(0, 3.2, 3.2), (100, 0.5, 0.5), (200, 0.2, 0.2)]
overfit = [(0, 3.2, 3.2), (100, 0.2, 0.36), (200, 0.1, 0.49), (300, 0.07, 0.62)]
stuck = [(0, 3.2, 3.2), (100, 3.0, 3.0), (200, 2.9, 2.9)]
blown = [(0, 3.2, 3.2), (50, 17.0, 16.8), (100, 19.0, 18.7)]
print([diagnose(h) for h in (healthy, overfit, stuck, blown)])   # ['healthy', 'overfitting', 'underfitting', 'diverged']

N = 124_412_160
D = 20 * N
print(f"{D:.2e} tokens, {6 * N * D:.2e} FLOPs")                   # 2.49e+09 tokens, 1.86e+18 FLOPs
print(round(10 ** -0.076, 3), round(10 ** -0.095, 3))              # 0.839 0.804

Engineering implications and limitations

Trust validation loss on clean data. Make sure the validation text is separated before windowing (Lesson LM2-T6) and deduplicated against training data. Otherwise the best-looking checkpoint may be the most memorised one.

Change one thing at a time. When a curve looks wrong, vary only the learning rate or only the model size per run, and compare on the same validation batches.

More data beats more epochs. Repeating a small data set many times invites memorisation. If data are scarce, a few epochs are fine; beyond a handful, returns drop quickly. Data augmentation is hard for text; better is more diverse text.

Plan with C ≈ 6ND. Before launching, estimate tokens and FLOPs, convert to GPU hours at a realistic utilisation (30–50% of peak), and check that the budget is available. Run a small version first to catch bugs.

Dedupe. Exact deduplication by hash is cheap; near-duplicate removal (MinHash and similar) takes more work and matters for web-scale data.

Be wary of benchmark leakage. If you evaluate on a public benchmark, check that its text does not appear in your training data; report if you could not.

Small models and data for robots. For a domain model (e.g. maintenance logs) the data are small, so overfitting is the main risk: start from a pretrained model and fine-tune (Modules 6–8) rather than pretraining from scratch.

Common misconceptions

Engineering takeaways

1. Healthy: both curves fall together. Overfitting: training falls, validation rises. Underfitting: both stall high. Divergence: the loss jumps or becomes NaN.
2. Memorisation and duplicated data inflate training performance without generalisation; validation must be separate and deduplicated.
3. Loss falls as a power law of parameters and data (exponents about 0.076 and 0.095 in Kaplan et al.); compute-optimal training uses roughly 20 tokens per parameter (Hoffmann et al.).
4. Training compute C ≈ 6ND FLOPs: about 1.9e18 for a 124-million-parameter model on 2.5 billion tokens.
5. Change one thing at a time and compare on the same validation batches.

References and further reading

See Chapter 5 of the book for training and validation losses and the discussion of overfitting. The scaling-law constants quoted here come from the papers cited and apply to the models and data those papers studied.

• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
• Kaplan et al. (2020), Scaling Laws for Neural Language Models (research paper)
• Build a Large Language Model (From Scratch), Chapter 5: Pretraining on Unlabeled Data (primary book reference)
• Build a Large Language Model (From Scratch), Appendix D: Adding Bells and Whistles to the Training Loop (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: generation. From logits to text with temperature and sampling.

Related content

Nothing below is required to finish this lesson.

• Before
  LM5-T3: Training Loops, Validation and Checkpointing

  Write a training loop with a validation pass, logging and checkpoints.

• Lab
  LM5-L3: Pretrain a Tiny GPT and Diagnose the Run

  Train a small GPT on a small corpus, plot training and validation loss, sample at different stages and diagnose a bad run.