5.5 — Generalization Across Tasks and Objects

Learning objectives

After this lesson, you should be able to:

• Name the three axes of generalisation.
• Explain why to perturb one axis at a time.
• Explain composition and when it works.
• List questions to ask of a generalisation claim.

The one-sentence idea

Say which kind of generalisation you mean, then test that kind.

Change one thing at a time.

Generalisation is not one thing

“It generalises” can mean very different things: a new object, a new position, a new phrasing, a new room, a new robot. Lumping them together makes claims impossible to compare. A recent taxonomy for robot manipulation sorts generalisation along three axes that follow the policy’s own inputs and outputs: visual (perturb the images), semantic (perturb the language) and behavioural (perturb the actions needed).^[1] The same work reports that current VLAs struggle with several kinds of semantic generalisation, even though they are pre-trained on Internet-scale language data.^[1]

That finding is a useful correction to the story in lessons 5.2 and 5.4. RT-2 reported better generalisation to novel objects and to commands absent from the robot data,^[2] but those are particular kinds of generalisation, evaluated on particular models. The limit stated in the RT-2 paper still holds: physical skills stay within the distribution of the robot data.^[2]

Testing one axis at a time

The practical lesson: change one thing at a time, so that a failure tells you where the weakness is. The toy below has a policy that sees a coloured object at some position and a one-word instruction, and picks it up. We then perturb the three axes separately (the policy is deliberately flawed in a different way for each, so the experiment can tell them apart):

import numpy as np
rng = np.random.default_rng(0)

colours = ["red", "green", "blue"]

def policy(scene, word):
    # semantic part: it only knows these three words
    if word not in colours:
        return None
    # visual part: it recognises objects by an exact colour value, learned under normal light
    candidates = [o for o in scene if abs(o["brightness"] - 1.0) < 0.2 and o["colour"] == word]
    if not candidates:
        return None
    # behavioural part: it can only reach objects within 0.75 of the origin
    pos = candidates[0]["pos"]
    return pos if np.linalg.norm(pos) <= 0.75 else None

def success(light=1.0, word_set=colours, spread=0.5, trials=500):
    wins = 0
    for _ in range(trials):
        scene = [{"colour": c, "brightness": light, "pos": rng.uniform(-spread, spread, 2)} for c in colours]
        word = word_set[rng.integers(len(word_set))]
        target = [o for o in scene if o["colour"] == word]
        act = policy(scene, word)
        wins += act is not None and bool(target) and np.allclose(act, target[0]["pos"])
    return wins / trials

print("as trained:                       ", success())
print("visual change (darker room):      ", success(light=0.6))
print("semantic change (new words):      ", success(word_set=["crimson", "emerald", "navy"]))
print("behavioural change (far objects): ", success(spread=1.0))

as trained:                        1.0
visual change (darker room):       0.0
semantic change (new words):       0.0
behavioural change (far objects):  0.424

Toy model. A hand-written policy with three planted weaknesses. It shows the testing method, one axis at a time; it says nothing about how any real VLA fails.

Each perturbation breaks the policy in its own way (one entirely, one partly), and the table tells you which part to fix. If you had changed everything at once, you would only know that it failed.

Combinations: the new pair from familiar parts

A common kind of generalisation is composition: never saw “push the cup” but did see “push the box” and “lift the cup”. A model that stores each (verb, object) pair separately cannot do it; a model that learns the verb and the object separately can. The toy trains both kinds on a subset of the pairs and tests on the unseen ones:

import numpy as np
rng = np.random.default_rng(0)

verbs, objects = 5, 5
verb_effect = rng.normal(size=(verbs, 2))                  # how each verb changes the action
object_effect = rng.normal(size=(objects, 2))               # how each object changes the action
def action(v, o): return verb_effect[v] + object_effect[o]

pairs = [(v, o) for v in range(verbs) for o in range(objects)]
order = rng.permutation(len(pairs))
train = [pairs[i] for i in order[:15]]                      # 15 of the 25 combinations are demonstrated
test  = [pairs[i] for i in order[15:]]                      # the other 10 are never seen

# model 1: lookup table of seen pairs (falls back to the average action if the pair is new)
table = {p: action(*p) for p in train}
mean_action = np.mean(list(table.values()), axis=0)
lookup_err = np.mean([np.linalg.norm(table.get(p, mean_action) - action(*p)) for p in test])

# model 2: additive model, one vector per verb and one per object, fitted by least squares
X = np.zeros((len(train), verbs + objects))
for i, (v, o) in enumerate(train):
    X[i, v] = 1; X[i, verbs + o] = 1
Y = np.array([action(*p) for p in train])
W = np.linalg.lstsq(X, Y, rcond=None)[0]
def predict(p):
    x = np.zeros(verbs + objects); x[p[0]] = 1; x[verbs + p[1]] = 1
    return x @ W
additive_err = np.mean([np.linalg.norm(predict(p) - action(*p)) for p in test])

print("error on unseen combinations, lookup table:", round(float(lookup_err), 3))
print("error on unseen combinations, additive parts:", round(float(additive_err), 3))

error on unseen combinations, lookup table: 0.958
error on unseen combinations, additive parts: 0.0

Toy model. The world is built to be additive, which is exactly what the second model assumes. Real tasks are rarely that tidy, so composition works only when the parts really combine.

The part-based model transfers to unseen combinations because its structure matches the world. Whether a neural VLA does something similar for a given combination is an empirical question, answered only by testing held-out combinations.

What to ask of a generalisation claim

When you read “generalises to new objects”, ask: which axis (visual, semantic, behavioural)? Compared with what baseline? How many trials (lesson 3.4)? And is the new case really absent from the training data? OpenVLA, for example, reports its comparison with RT-2-X as 16.5 percentage points of absolute task success across 29 tasks and several robot embodiments,^[3] a statement with a defined unit, a defined set of tasks and a defined baseline. That is the form to expect, and to ask for.

Common misconceptions

Engineering takeaways

1. Three axes: visual, semantic, behavioural.
2. Perturb one at a time.
3. Composition needs structure.
4. Ask about baselines, trials and training-data overlap.

Generalisation to what, exactly?

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the foundational research. Next: the modern VLA models, starting with OpenVLA.

Back to the VLA roadmap