M3.5 — Pretraining and Fine-Tuning

Learning objectives

After this lesson, you should be able to:

• Explain pretraining and fine-tuning and why the combination is used.
• Describe next-token, masked and contrastive pretraining objectives.
• Compute a cross-entropy loss for a single prediction.
• Show, with a small example, why starting from a pretrained model helps.
• Explain freezing and low-rank adaptation.
• Name the main risks: forgetting, distribution shift and leakage.

The one-sentence idea

Pretrain a model on abundant general data to learn broad features, then fine-tune it on scarce task data so it adapts instead of starting from scratch.

Learn broadly first, then specialise.

Two stages: learn broadly, then specialise

Training a large model from scratch needs an enormous amount of data. Robot data is scarce: every demonstration needs a real robot and a person to operate it. Text and images from the internet are plentiful. Pretraining and fine-tuning bridge the gap:

1. Pretraining. Train a large model on a huge, general dataset with a simple objective, so it learns broadly useful features: what objects look like, how language is structured, how the two relate.
2. Fine-tuning. Continue training the same model on a much smaller dataset for the specific task, so it adapts what it already knows instead of learning everything again.

This is the pattern behind most VLA models: a vision-language model pretrained on web-scale image and text data is then trained, in some form, on robot demonstrations so that it can output actions. The roadmap’s later phases look at how specific models do this. This lesson covers the general idea.

Pretraining objectives

Pretraining uses objectives that need no human labels, because the data supplies its own answer. Three are especially common:

Next-token prediction

Given the tokens so far, predict the next one. The answer is simply the following token in the text. This trains the decoder-style language models of M3.4.

Masked prediction

Hide some tokens or image patches and predict them from the rest. The answer is the hidden piece.

Contrastive learning

Pull the embeddings of matching pairs, such as an image and its caption, together, and push mismatched pairs apart. This is how CLIP is trained.

For next-token prediction the loss is the cross-entropy: the negative log of the probability the model gave to the correct token. A confident right answer costs little and a confident wrong one costs a lot:

import numpy as np

print(round(-np.log(0.7), 3))     # the model gave the correct token probability 0.7
print(round(-np.log(0.05), 3))    # it gave the correct token only 0.05

0.357
2.996

The contrastive objective can be sketched with the dot-product similarity of M1.1. Take three images and their three captions as vectors. If each caption is most similar to its own image, the softmax over similarities puts high probability on the correct pairing and the loss is small. If the captions are shuffled, the loss is large:

def contrastive_loss(images, texts, temperature=0.5):
    images = images / np.linalg.norm(images, axis=1, keepdims=True)
    texts = texts / np.linalg.norm(texts, axis=1, keepdims=True)
    sims = images @ texts.T / temperature                      # (n, n) similarity table
    e = np.exp(sims - sims.max(axis=1, keepdims=True))
    p = e / e.sum(axis=1, keepdims=True)                       # softmax over the captions
    return -np.log(np.diag(p)).mean()                          # reward the matching caption

images = np.eye(3)                                    # three image embeddings (hand-made)
texts = np.array([[0.9, 0.1, 0.0],
                  [0.0, 0.9, 0.1],
                  [0.1, 0.0, 0.9]])                   # each caption is close to its own image
print(round(contrastive_loss(images, texts), 3))
print(round(contrastive_loss(images, texts[[1, 2, 0]]), 3))     # captions shuffled

0.268
2.256

The vectors here are made by hand to show the effect. In real training, the encoders are adjusted so that matched pairs end up close together across millions of examples, and the resulting embeddings are what M4 later calls vision-language alignment.

Fine-tuning: adapting what the model already knows

Fine-tuning starts from the pretrained parameters instead of random ones and continues training on the target data, typically with a smaller learning rate so that it does not overwrite everything it learned. The effect is easiest to see on a toy problem. First, “pretrain” a one-weight model on plenty of data from the line y = 2x + 1. Then give both that model and a fresh, untrained one only four examples from a slightly different task, y = 2x + 1.5, and train each for five steps:

import torch
from torch import nn

def make_model(w, b):
    m = nn.Linear(1, 1)
    with torch.no_grad():
        m.weight.fill_(w)
        m.bias.fill_(b)
    return m

def fit(model, x, y, steps, lr=0.1):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    loss_fn = nn.MSELoss()
    for _ in range(steps):
        loss = loss_fn(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
    return loss_fn(model(x), y).item()

x_big = torch.linspace(-1, 1, 20).unsqueeze(1)
pretrained = make_model(0.0, 0.0)
fit(pretrained, x_big, 2 * x_big + 1, 200)                 # pretraining on plenty of data

x_new = torch.tensor([[-1.0], [-0.3], [0.4], [1.0]])       # only four examples
y_new = 2 * x_new + 1.5                                    # a slightly different task

scratch = make_model(0.0, 0.0)
print("before:", round(fit(pretrained, x_new, y_new, 0), 3), round(fit(scratch, x_new, y_new, 0), 3))
print("after 5 steps:", round(fit(pretrained, x_new, y_new, 5), 3), round(fit(scratch, x_new, y_new, 5), 3))

before: 0.25 4.65
after 5 steps: 0.027 0.902

The pretrained model starts close to the answer, since the two tasks share most of their structure, so its loss after five steps (0.027) is far lower than that of the model that started from scratch (0.902). Real pretrained models give the same kind of head start, on tasks with far more structure to share. The same logic is why a few hundred robot demonstrations can be enough to fine-tune a model whose perception and language understanding were learned elsewhere.

Freezing and parameter-efficient fine-tuning

Updating every parameter of a very large model is costly in memory and risks damaging what it learned. Two common alternatives:

• Freezing. Keep some layers fixed and train only the rest, often a small head on top. In PyTorch, set requires_grad = False on the frozen parameters.
• Low-rank adaptation (LoRA). Leave the original weight matrix W untouched and learn a small correction A × B, where A and B are thin matrices, so only a small fraction of the parameters changes.

net = nn.Sequential(nn.Linear(1, 8), nn.ReLU(), nn.Linear(8, 1))
for p in net[0].parameters():
    p.requires_grad = False                 # freeze the first layer
trainable = sum(p.numel() for p in net.parameters() if p.requires_grad)
total = sum(p.numel() for p in net.parameters())
print(trainable, total)

# the idea behind LoRA: a correction A @ B with rank 8 instead of a full 1000 x 1000 update
full = 1000 * 1000
low_rank = 1000 * 8 + 8 * 1000
print(full, low_rank, round(100 * low_rank / full, 1))

9 25
1000000 16000 1.6

Freezing the first layer leaves 9 of the 25 parameters trainable. A rank-8 correction to a 1000 × 1000 matrix needs 16,000 numbers instead of a million, which is 1.6% of the full update. The idea uses the matrix shapes of M1.1: a (1000, 8) matrix times an (8, 1000) matrix is a (1000, 1000) matrix.

What can go wrong

• Forgetting. Fine-tuning too long or too aggressively can erase useful general knowledge. A low learning rate, freezing, or mixing in some of the original data helps.
• A mismatch between pretraining and target. If the target is very different, as robot sensor streams are from web photos, the pretrained features help less. This gap is called distribution shift, and it is the subject of M6.
• Leakage. If test examples were part of the huge pretraining data, evaluation results are inflated. Check what a model has seen.
• Pretraining is not magic. It supplies general features. The specific behaviour, such as controlling a particular robot, still has to come from the fine-tuning data.

Common misconceptions

Engineering takeaways

1. Pretraining learns general features from large, cheap data. Fine-tuning adapts them with small, costly data.
2. Common objectives are next-token prediction, masked prediction and contrastive learning.
3. Cross-entropy is the negative log of the probability given to the correct answer.
4. A pretrained start gives a much lower loss after few steps than starting from scratch, when the tasks are related.
5. Freezing trains only some parameters, and LoRA learns a small low-rank correction.
6. Forgetting, distribution shift and test-data leakage are the main risks.
7. A VLA starts from a vision-language model and is adapted using robot data.
8. The pretrained model supplies features. The robot behaviour still comes from the robot data.

Pretraining supplies what the model knows about the world, and fine-tuning teaches it the job.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the deep-learning and transformer prerequisites. Next come computer vision and language-model foundations.

Back to the VLA roadmap prerequisites