5.4 — Vision-Language Pretraining and Robot Demonstration Data

Learning objectives

After this lesson, you should be able to:

• Name the two data sources and what each teaches.
• Contrast pretrain-then-fine-tune and co-fine-tuning.
• Explain why pretraining reduces the demonstrations needed.
• Explain forgetting and how mixing helps.

The one-sentence idea

Web data supply knowledge; robot data supply skills.

Start from knowledge, add skill, and do not throw the knowledge away.

Two kinds of data, in stages

A modern VLA is trained from two very different sources. Web-scale data (images with captions, questions about pictures, text) is abundant and teaches general knowledge. Robot demonstrations are scarce and teach how to move. The recipes combine them in different ways:

• Pretrained parts, then robot training. OpenVLA is built on a Llama 2 language model with a visual encoder that fuses pretrained DINOv2 and SigLIP features, and is then trained on 970,000 real-world robot demonstrations.^[1] π0 builds on a pretrained vision-language model, PaliGemma, which is about 3 billion parameters made from a SigLIP image encoder and the Gemma-2B language model.^[2][3]
• Co-fine-tuning. RT-2 fine-tunes vision-language models on robot trajectories and Internet-scale vision-language tasks at the same time.^[4]

The rest of the lesson gives two small experiments for why these recipes are used.

Why start from a pretrained model: fewer demonstrations needed

Robot demonstrations are expensive (lesson 5.1). A pretrained encoder has already learned which aspects of the input matter and which are noise. In the toy below, the input has 50 numbers but the action depends on only 2 of them; the other 48 are distractions. A model that must discover this from demonstrations alone needs many; one that starts with the useful features needs few:

import numpy as np
rng = np.random.default_rng(0)

d, relevant = 50, 2
w_true = np.zeros(d); w_true[:relevant] = [1.0, -2.0]               # only the first two inputs matter

def data(n):
    X = rng.normal(size=(n, d))
    return X, X @ w_true + rng.normal(0, 0.1, n)

def ridge(X, y, lam=1e-3):
    return np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ y)

Xt, yt = data(5000)
print("demos | from raw inputs | from pretrained features")
for n in (10, 30, 100, 400):
    X, y = data(n)
    w_raw = ridge(X, y)                                              # learns from all 50 numbers
    w_pre = np.zeros(d); w_pre[:relevant] = ridge(X[:, :relevant], y)   # "pretrained": already looks at the right 2
    e_raw = np.mean((Xt @ w_raw - yt) ** 2); e_pre = np.mean((Xt @ w_pre - yt) ** 2)
    print(f"{n:>5} | {e_raw:15.4f} | {e_pre:24.4f}")

demos | from raw inputs | from pretrained features
   10 |          4.6629 |                   0.0113
   30 |          1.3792 |                   0.0102
  100 |          0.0155 |                   0.0099
  400 |          0.0114 |                   0.0100

Toy model. The pretrained features are handed over exactly. Real pretrained features are imperfect and learned from images and text, but the benefit is of this kind: fewer things to learn from scarce robot data.

With 10 demonstrations the raw model is far off, while the one with good features is already near the noise floor (0.01). With enough data they converge. That is the economic argument for pretraining: it buys down the number of demonstrations needed.

Why co-train: do not forget the web

Fine-tuning on robot data alone pulls the model’s parameters toward what the robot task needs, and away from what the web tasks needed. The general knowledge you wanted to keep can fade, a failure known as forgetting. Mixing web and robot examples during fine-tuning keeps both objectives pulling. A toy with one shared set of weights shows the effect:

import numpy as np
rng = np.random.default_rng(0)

d = 5
w_web   = np.array([1.0, 1.0, 1.0, 0.0, 0.0])                       # what the "web task" wants
w_robot = np.array([1.0, 1.0, 0.0, 1.0, 1.0])                       # what the "robot task" wants (shares two directions)

def batch(w, n=32):
    X = rng.normal(size=(n, d)); return X, X @ w

def train(mix_web, steps=2000, lr=0.01, start=None):
    w = w_web.copy() if start is None else start.copy()              # begin from the "pretrained" model, good at the web task
    for _ in range(steps):
        Xr, yr = batch(w_robot)
        g = Xr.T @ (Xr @ w - yr) / len(yr)
        if mix_web:
            Xw, yw = batch(w_web)
            g = g + Xw.T @ (Xw @ w - yw) / len(yw)
        w -= lr * g
    return w

def loss(w, target):
    X, y = batch(target, 4000); return float(np.mean((X @ w - y) ** 2))

for name, mix in (("robot data only", False), ("robot + web mixed", True)):
    w = train(mix)
    print(f"{name:<18} robot loss {loss(w, w_robot):.3f} | web loss {loss(w, w_web):.3f}")

robot data only    robot loss 0.000 | web loss 2.889
robot + web mixed  robot loss 0.712 | web loss 0.787

Toy model. Linear models and two overlapping tasks. In a toy with enough capacity both could be learned; the point is only the direction of the effect.

Robot-only fine-tuning does best on the robot task and loses the web task. Mixed fine-tuning compromises: a worse robot loss (0.71 instead of 0.00) in exchange for a much better web loss (0.79 instead of 2.89). In this toy the two tasks conflict, so you cannot have both fully; real models have far more capacity. RT-2 reports that its co-fine-tuned models show better generalisation to novel objects and follow commands absent from the robot data.^[4]

What to remember

Pretraining and robot data play different parts: pretraining supplies knowledge and good features; robot data supply the skills. Remember the limit from lesson 5.2: physical skills stay within the distribution of the robot data.^[4] A bigger pretrained model does not remove the need for the right demonstrations.

Common misconceptions

Engineering takeaways

1. Pretraining = knowledge, robot data = skill.
2. Good features cut demonstration needs.
3. Co-training protects web skills.
4. Skills still need demonstrations.

Knowledge from the web, skill from the robot.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: what generalisation means for robot policies and how it is demonstrated.

Back to the VLA roadmap