LM6-T1 — Pretraining Versus Supervised Fine-Tuning

Learning objectives

After this lesson, you should be able to:

• Contrast pretraining and supervised fine-tuning by data, objective, cost and output.
• Explain transfer learning and why fine-tuning needs far fewer labelled examples than training from scratch.
• Estimate the relative compute of fine-tuning and pretraining.
• List the risks: forgetting, overfitting small data, and label problems.

The idea: learn the language once, the task cheaply

Pretraining uses text that needs no human labels, because the next token is its own label. The model absorbs spelling, grammar, facts and patterns of reasoning from billions of tokens. That is expensive but done once. Supervised fine-tuning then takes this general model and continues training on a small set of examples that each carry a label produced by people: a message and the word “spam”, a question and a good answer. The model does not need to learn language again; it needs to learn what the labels mean in terms of the representations it already has.

Why it works: transfer

The vectors a pretrained model computes for a text already separate many of the distinctions that matter for tasks: topic, sentiment, whether a sentence is a question, who did what. A small amount of labelled data is enough to find the directions in this space that matter for the label. Trained from scratch on the same few examples, a model would have to discover language and the task together and would usually overfit. This is called transfer learning, and it is the reason that a few thousand labelled examples can yield a strong classifier.

Label efficiency

Typical numbers: classification fine-tuning of a pretrained language model often reaches good accuracy with hundreds to a few thousand labelled examples, where training from scratch might need hundreds of thousands. The gain depends on how related the task is to the pretraining data. In the notebook the pretraining is a few seconds on a toy language and the task is trivial, so the advantage hardly shows; with a real pretrained model on a real task it is large.

What changes in training

• Starting point: pretrained weights, not random ones. New parts (a classification head) start random.
• Objective: cross-entropy against the task label, not against the next token (Lesson LM6-T2).
• Learning rate: smaller, typically 10⁻⁵ to 5·10⁻⁵ for the whole model, so that updates refine the weights and do not wipe out what pretraining learned.
• Duration: a few epochs over a small set, minutes to hours.

Risks

Catastrophic forgetting: training hard on a narrow task can erase general abilities. Overfitting: with few examples a large model memorises them (Lesson LM5-T4); validation data are essential. Label problems: wrong, inconsistent or biased labels teach the model the wrong thing, and the model will reproduce the bias (Lesson LM6-T4 and LM6-T5). Cost of change: a fine-tuned model is a new artifact to evaluate, version and maintain.

The mathematics: two objectives and their cost

Pretraining objective (Lesson LM5-T1): minimise the next-token loss over a large unlabelled corpus,

Fine-tuning objective: given examples (x, y) with a text x and a label y ∈ {1, …, K}, minimise the cross-entropy of the classifier’s distribution over the K labels,

θ

the pretrained parameters of the backbone; they are the starting point and may be updated or frozen.

φ

the new parameters (for example a classification head), initialised randomly.

M

the number of labelled examples (a batch average in practice).

q(y | x)

the classifier’s probability for label y given the text; a softmax over K logits.

The mathematical form is the same as in pretraining (cross-entropy), with a different set of outputs: K classes instead of V tokens. Only the target and the head change.

Cost. Training compute is about 6 · P · D FLOPs (Lesson LM5-T4) for P parameters and D tokens processed. For GPT-2 small, P = 124,412,160: pretraining on 2.5·10⁹ tokens costs about 1.9·10¹⁸ FLOPs. Fine-tuning on 5,000 examples of 100 tokens for 3 epochs processes D = 1.5·10⁶ tokens: 6 · 1.244·10⁸ · 1.5·10⁶ = 1.1·10¹⁵ FLOPs, about 1,660 times less. On a GPU at 10¹⁴ FLOP/s that is seconds of arithmetic and minutes in practice.

Head size. The vocabulary head has d · V parameters (38,597,376 for GPT-2 small). A classification head with K classes has d · K + K: 1,538 for K = 2.

A worked example: swapping the head

The change from a language model to a classifier is the last layer. The code replaces the output head of a model with the width of GPT-2 small (the backbone is a stand-in with random weights, to keep the example light) and counts the parameters.

import torch
import torch.nn as nn

d, V, K = 768, 50257, 2

lm_head = nn.Linear(d, V, bias=False)           # pretraining: one score per vocabulary entry
cls_head = nn.Linear(d, K)                      # fine-tuning: one score per class
print(sum(p.numel() for p in lm_head.parameters()))     # 38597376
print(sum(p.numel() for p in cls_head.parameters()))    # 1538

# the hidden state of one position goes to either head
h = torch.randn(4, d)                           # 4 sequences, one pooled state each
print(tuple(lm_head(h).shape), tuple(cls_head(h).shape))   # (4, 50257) (4, 2)

# loss: cross-entropy over the classes
y = torch.tensor([0, 1, 1, 0])
loss = torch.nn.functional.cross_entropy(cls_head(h), y)
print(round(loss.item(), 2))                    # near ln 2 = 0.69 for a head that knows nothing

# relative compute: pretraining versus a typical fine-tuning run
pre = 6 * 124_412_160 * 2.5e9
ft = 6 * 124_412_160 * (5000 * 100 * 3)
print(f"{pre:.1e} vs {ft:.1e} FLOPs, ratio {pre / ft:.0f}")   # 1.9e+18 vs 1.1e+15 FLOPs, ratio 1667

The ratio printed is 1,667 because the code uses 2.5·10⁹ tokens exactly; with 2.49·10⁹ (20 tokens per parameter) it is 1,661. Both say: fine-tuning costs about a thousandth of pretraining.

Which to choose

Engineering implications and limitations

Use a small learning rate for pretrained weights. A rate suited to training from scratch (10⁻³) can destroy a pretrained model in a few steps. 10⁻⁵ to 5·10⁻⁵ is typical for full fine-tuning; the new head can use a larger one.

Keep the tokenizer. The tokenizer, vocabulary and special tokens of the pretrained model must be used unchanged for fine-tuning and inference (Lesson LM2-T4).

Hold out data before anything else. With small datasets a validation set of a few hundred examples is the only protection against fooling yourself. Split first, tune second, test once.

Start with the simplest baseline. A bag-of-words logistic regression often gets close on easy tasks. If your fine-tuned model only matches it, the language model is not earning its cost.

Track what you start from. Record the exact checkpoint, data version, seed and hyperparameters. A fine-tuned model without this record cannot be reproduced or audited.

Check general ability afterwards. Evaluate on a few tasks you do not train on, to see whether fine-tuning damaged general behaviour (forgetting). Parameter-efficient methods (Module 8) reduce this risk by changing little.

Mind the licence. Pretrained weights and datasets have licences; check that your use is allowed before you fine-tune and release.

Common misconceptions

Engineering takeaways

1. Pretraining: unlabelled text, next-token loss, enormous compute. Fine-tuning: labelled examples, task loss, a tiny fraction of the compute.
2. Transfer learning lets a few hundred to a few thousand labelled examples suffice, because the backbone already represents language.
3. The loss form is the same cross-entropy; only the outputs (K classes instead of V tokens) and the data change.
4. Fine-tuning GPT-2 small on 1.5 million tokens costs about 1.1e15 FLOPs, roughly 1,660 times less than pretraining it.
5. Risks: catastrophic forgetting, overfitting small data, and labelling errors; use a small learning rate and a held-out set.

References and further reading

See Chapter 6 of the book (finetuning for classification: different categories of finetuning, and preparing the dataset).

• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI technical report.
• Build a Large Language Model (From Scratch), Chapter 6: Finetuning for Classification (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the classification head and which token’s hidden state represents the whole text.

Related content

Nothing below is required to finish this lesson.

• Before
  LM5-T6: Top-k, Top-p and Stopping Criteria

  Truncate the distribution with top-k and top-p, and decide when generation stops.

• Lab
  LM6-L1: Classification Metrics and Failure Cases

  Compute a confusion matrix, precision, recall and F1, and find the examples a classifier gets wrong for the wrong reason.