VLA · Phase 8 · VLA Evaluation and Generalization
8.3 — Generalization to New Objects, Positions and Instructions
Testing how far a policy holds up when the setup changes.
Generalisation is tested, not claimed. This lesson shows how to design a grid of single and combined changes, read the gap per factor and test whether instructions survive rewording.
Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.
Learning objectives
After this lesson, you should be able to:
- Design a test grid with one factor changed at a time.
- Compute and read the generalisation gap.
- Test instruction paraphrases.
- Report a generalisation test completely.
- Fix a control condition
- Change one factor at a time
- Add a combined condition
- Run enough trials in each
- Report each gap with the control
The one-sentence idea
Change one thing at a time and measure the drop.
Each factor gets its own gap.
Design the test before the policy
A generalisation test asks: how does success change when I change something the policy was not trained on? Lesson 5.5 sorted such changes into visual, semantic and behavioural axes.[1] This lesson turns that into a concrete test design for the three changes the roadmap names: new objects, new positions and new instructions. The key principle is a grid: vary one factor at a time while the others stay at their trained values, plus a small number of combined changes at the end.
trained = {"object": ["cup", "box"], "position": ["left", "centre"], "phrase": ["pick up the {}", "grab the {}"]}
held_out = {"object": ["bottle"], "position": ["far right"], "phrase": ["could you lift the {}"]}
tests = [("in distribution", {k: v[0] for k, v in trained.items()})]
for factor in trained:
config = {k: v[0] for k, v in trained.items()}
config[factor] = held_out[factor][0]
tests.append((f"new {factor}", config))
tests.append(("all three new", {k: v[0] for k, v in held_out.items()}))
for name, cfg in tests:
print(f"{name:<17}", cfg)Output:
in distribution {'object': 'cup', 'position': 'left', 'phrase': 'pick up the {}'}
new object {'object': 'bottle', 'position': 'left', 'phrase': 'pick up the {}'}
new position {'object': 'cup', 'position': 'far right', 'phrase': 'pick up the {}'}
new phrase {'object': 'cup', 'position': 'left', 'phrase': 'could you lift the {}'}
all three new {'object': 'bottle', 'position': 'far right', 'phrase': 'could you lift the {}'}Five test conditions: the control, three single changes, and everything changed at once. Run many trials of each (lesson 8.2) and report all five.
Read the generalisation gap
The generalisation gap is the drop in success from the control to a changed condition. A policy with a small gap on one factor and a big gap on another has a specific weakness. For the toy policy below, only the object is recognised from appearance, the position is computed geometrically, and the instruction is matched by exact phrase:
import numpy as np
rng = np.random.default_rng(0)
known_objects = {"cup", "box"}
known_phrases = {"pick up the", "grab the"}
def attempt(obj, position_x, phrase):
if obj not in known_objects: return False # cannot recognise a new object
if phrase not in known_phrases: return False # only exact phrases understood
reach = 1.0 - 0.15 * abs(position_x) # success gets harder far from the centre
return rng.random() < reach
conditions = {
"in distribution": ("cup", 0.3, "pick up the"),
"new object": ("bottle", 0.3, "pick up the"),
"new position": ("cup", 3.0, "pick up the"),
"new phrase": ("cup", 0.3, "could you lift the"),
}
control = None
for name, (obj, x, phrase) in conditions.items():
rate = np.mean([attempt(obj, x, phrase) for _ in range(400)])
control = rate if control is None else control
print(f"{name:<16} success {rate:.2f} gap {control - rate:+.2f}")Output:
in distribution success 0.94 gap +0.00 new object success 0.00 gap +0.94 new position success 0.54 gap +0.40 new phrase success 0.00 gap +0.94
Toy model. A hand-written policy with deliberate weaknesses, as in lesson 5.5. It shows how to read the table, not how any real VLA behaves.
The table says what to fix: object recognition and instruction understanding are all-or-nothing here, while position degrades gradually. A single overall success number would have hidden all of that.
Instructions: paraphrases are a test of their own
People do not repeat the words used in training. A STAR-Gen study reports that current VLAs struggle with several kinds of semantic generalisation even though they are pre-trained on Internet-scale language data.[1] You can test this in your own system by writing paraphrases of each training instruction and measuring success on them. Three ways a policy might read the words, compared on paraphrases:
import numpy as np
train_phrases = {"pick up the red block": "red", "pick up the blue block": "blue"}
paraphrases = ["grab the red block", "please lift the red cube", "take the blue block", "I want the blue one", "move the green block"]
truth = ["red", "red", "blue", "blue", "green"]
def exact(text): return train_phrases.get(text)
def keyword(text):
for colour in ("red", "blue"):
if colour in text.split(): return colour
return None
vocab = {"red": np.array([1.0, 0.0]), "blue": np.array([0.0, 1.0]), "green": np.array([0.7, 0.7])} # a (tiny) shared meaning space
def similarity(text):
words = [w for w in text.split() if w in vocab]
if not words: return None
v = vocab[words[0]]
best = max(("red", "blue"), key=lambda c: float(v @ vocab[c] / (np.linalg.norm(v) * np.linalg.norm(vocab[c]))))
return best
for name, reader in (("exact match", exact), ("keyword spotting", keyword), ("similarity", similarity)):
answers = [reader(t) for t in paraphrases]
correct = sum(a == t for a, t in zip(answers, truth) if t in ("red", "blue"))
print(f"{name:<17} understands {correct} of 4 paraphrases of known colours; answers for the unknown 'green': {answers[-1]}")Output:
exact match understands 0 of 4 paraphrases of known colours; answers for the unknown 'green': None keyword spotting understands 4 of 4 paraphrases of known colours; answers for the unknown 'green': None similarity understands 4 of 4 paraphrases of known colours; answers for the unknown 'green': red
Exact matching understands none of the paraphrases. Keyword spotting handles all four because each contains a trained colour word, but it fails on the new word. The similarity reader also copes with the new word, though “green” was mapped to “red”, its closest known colour: an answer, but not necessarily a right one, which is why such behaviour must be tested, not assumed. Real language models behave like the third, but only as well as their training allows, which is what the semantic tests measure.
Reporting a generalisation test
A good report names each condition, how many trials it had, which factor changed, and what was held out and why. It also states whether the changed value truly never appeared in the training data, because an unseen object in the test set that appeared in the pre-training web data is a different claim from one that did not. Finally, report the control: a gap needs a baseline to be a gap.
Common misconceptions
Misconception 1 “Testing on new scenes tests generalisation.”
Correction Only if you know which factors changed.
Misconception 2 “If it understands the training phrase it understands the idea.”
Correction Paraphrases test the language separately.
Misconception 3 “Unseen in the robot data means unseen.”
Correction It may be in the pre-training data; say which.
Engineering takeaways
- Control + single changes + combined change.
- Gap = control minus condition.
- Test paraphrases.
- Report trials and training overlap.
No control, no gap.
Knowledge check
Try each question first, then reveal the explanation.
Question 1
Why include a combined-change condition?
Explanation
Failures can compound; single changes may hide that.
Question 2
Control 0.90, new object 0.40. What is the gap?
Explanation
0.50.
Question 3
Why can a new object still be “seen” by the model?
Explanation
It may have appeared in pre-training data.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2025). A taxonomy for evaluating generalist robot policies. arXiv; later versions retitled “…generalist robot manipulation policies” (IEEE RA-L). arXiv:2503.01238
Cited for:
- Proposes a taxonomy for generalisation in robot manipulation with three axes that follow the policy’s input and output modalities: visual (perturb the images), semantic (perturb the language) and behavioural (perturb the actions). Checked against the paper’s abstract.
- Reports that current vision-language-action models struggle with several kinds of semantic generalisation even though they are pre-trained on Internet-scale language data. Checked against the paper’s abstract.
What’s next
Next: telling apart perception, grounding and action failures.