11.2 — VLA Fine-Tuning and Data Efficiency

Learning objectives

After this lesson, you should be able to:

• Compute the parameter saving of low-rank adaptation.
• Show how noisy copies regularise a small-data fit.
• Compare targeted and random demonstration collection.
• List other data sources and their gaps.

The one-sentence idea

Spend scarce data where it counts.

Less to learn, more from each example, better-chosen examples.

Data is the bottleneck

The data behind robot models is costly: RT-1's dataset took 13 robots 17 months to collect,^[1] and even a strong adaptation result, such as Gemini Robotics learning new short tasks from as few as 100 demonstrations, costs hours of operator time (lesson 10.2).^[2] So research on data efficiency asks: how do we get good behaviour from less robot data? The main levers, each with a small experiment below: adapt only a few parameters, augment the data you have, choose which data to collect, and use other data sources.

Lever 1: change fewer parameters

Low-rank adaptation freezes the pretrained weights and adds small trainable low-rank matrices to the Transformer layers. Against GPT-3 175B fine-tuned with Adam, the LoRA authors report about 10,000 times fewer trainable parameters and about 3 times less GPU memory, with quality on par or better, and no additional inference latency.^[3] OpenVLA's authors likewise report fine-tuning on consumer GPUs via low-rank adaptation without a hit to downstream success rate.^[4] Fewer trainable parameters also means less capacity to memorise a small dataset. The parameter arithmetic, as in lesson 6.1, for a stack of layers:

layers, d, r = 32, 4096, 8                     # an example stack: 32 matrices of 4096 x 4096, rank 8
full = layers * d * d
lora = layers * 2 * d * r
print(f"full fine-tuning of these matrices: {full:,} trainable parameters")
print(f"low-rank adaptation (rank {r}):       {lora:,} trainable parameters")
print(f"ratio: {full / lora:,.0f} times fewer")

full fine-tuning of these matrices: 536,870,912 trainable parameters
low-rank adaptation (rank 8):       2,097,152 trainable parameters
ratio: 256 times fewer

Toy model. An example size, not a particular model. The ratio d / (2r) does not depend on the number of layers.

The ratio is d/(2r), 256 here. Larger models with the same rank give larger ratios, which is how the paper reaches 10,000× on a 175B model.

Lever 2: make each demonstration count more with augmentation

Augmentation creates modified copies of demonstrations (shifted crops, colour jitter, lighting changes) that the policy should treat the same way. It teaches the model what does not matter. Consider 15 demonstrations that were all recorded under the same lighting: a model cannot tell lighting from content, and when the lights change at test time it fails (lesson 3.5). In the toy below, a “lighting” direction is added to every input; the action depends only on the underlying scene. We add copies of each demonstration with random lighting and refit:

import numpy as np
rng = np.random.default_rng(0)

d, k, n = 30, 3, 15                                         # input size, hidden scene numbers, demonstrations
mix = rng.normal(size=(k, d))                               # how the scene shows up in the input
light = rng.normal(size=d); light /= np.linalg.norm(light)  # the direction in which lighting changes the input
w = np.array([1.0, -1.0, 0.5])                              # how the action depends on the scene

def data(count, light_sd):
    scene = rng.normal(size=(count, k))
    L = rng.normal(0, light_sd, count)
    X = scene @ mix + 2.0 * np.outer(L, light) + rng.normal(0, 0.1, (count, d))
    return X, scene @ w + rng.normal(0, 0.1, count)

def fit(X, y, lam=1e-3): return np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)
def test_error(weights):
    X, y = data(5000, light_sd=3.0)                         # test: the lighting varies a lot
    return float(np.mean((X @ weights - y) ** 2))

plain, augmented = [], []
for _ in range(40):                                         # many random small datasets
    X, y = data(n, light_sd=0.0)                            # all demonstrations recorded under the same lighting
    plain.append(test_error(fit(X, y)))
    Xa = np.vstack([X + 2.0 * np.outer(rng.normal(0, 3.0, len(X)), light) for _ in range(30)])   # copies with random lighting
    augmented.append(test_error(fit(Xa, np.tile(y, 30))))
print("trained on the 15 demonstrations only:       test error", round(float(np.mean(plain)), 3))
print("trained with random-lighting copies as well:  test error", round(float(np.mean(augmented)), 3))

trained on the 15 demonstrations only:       test error 1.464
trained with random-lighting copies as well:  test error 0.024

Toy model. A linear model with a lighting direction added by hand. Real augmentations act on images, and must reflect the variation the robot will really meet.

The augmented fit ignores lighting because it saw it vary without the action changing; the plain fit may have used it. An augmentation that breaks the task, such as flipping an image when left and right matter, would hurt, so choose augmentations that change nothing about the correct action.

Lever 3: choose which demonstrations to collect

Demonstrations are not equal. Another demonstration of something the data already covers adds little; one in an uncovered region adds a lot (lesson 5.1 showed this with goal variety). A simple strategy places the next demonstration where the current data is sparsest. Compare with picking random places, for a policy that is a nearest-neighbour lookup over the task space:

import numpy as np
rng = np.random.default_rng(0)

def target(x): return np.sin(3 * x)                                  # what the demonstrator would do at position x
grid = np.linspace(0, 2, 400)

def test_error(xs):
    xs = np.array(xs); ys = target(xs)
    pred = np.array([ys[np.argmin(np.abs(xs - g))] for g in grid])    # nearest-neighbour policy
    return float(np.mean(np.abs(pred - target(grid))))

def sparse_choice(xs):
    gaps = [(np.min(np.abs(np.array(xs) - g)), g) for g in grid]       # the position farthest from any collected demo
    return max(gaps)[1]

random_err, targeted_err = [], []
for trial in range(30):
    start = list(rng.uniform(0, 2, 3))
    r, t = list(start), list(start)
    for _ in range(9):
        r.append(rng.uniform(0, 2)); t.append(sparse_choice(t))
    random_err.append(test_error(r)); targeted_err.append(test_error(t))
print("12 demonstrations, error with random positions:   ", round(float(np.mean(random_err)), 3))
print("12 demonstrations, error with positions in gaps:  ", round(float(np.mean(targeted_err)), 3))

12 demonstrations, error with random positions:    0.147
12 demonstrations, error with positions in gaps:   0.087

Spreading demonstrations out gives lower error for the same count. In practice “where the data is sparse” is judged on task variations (objects, positions, lighting), and where a trained policy fails (lesson 7.6) is a good guide to what to collect next, which is the logic behind DAgger-style correction data (lesson 3.3).^[5]

Lever 4: use other data

Robot data can be supplemented: web data via co-training or pretrained parts (lesson 5.4), data pooled from other robots (lesson 5.3, with harmonisation), human videos and synthetic data in the humanoid models (lesson 10.1),^[6] and simulation (lesson 7.7). Each has its own gaps: web data teaches knowledge, not motion; other robots' data needs a shared action layout (lesson 6.5); synthetic data inherits the reality gap.^[7] The practical rule: get the pretrained model closest to your task, adapt with few parameters, augment sensibly, collect the demonstrations that fill gaps, and measure the improvement (lesson 8.2) before collecting more.

Common misconceptions

Engineering takeaways

1. d/(2r) fewer parameters.
2. Augmentation regularises.
3. Fill gaps.
4. Use other data with care.

Make every demonstration count.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: transferring one policy across robots with different bodies.

Back to the VLA roadmap