VLA · Phase 7 · VLA Implementation and Experimentation
7.6 — Model Evaluation and Failure Analysis
Measuring results and sorting failures into causes you can act on.
A success rate does not say what to fix. This lesson turns failures into labelled causes, finds where they cluster, and checks whether an improvement is real.
Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.
Learning objectives
After this lesson, you should be able to:
- Log facts that let you label failure causes.
- Count failures by cause.
- Slice success by condition and report trial counts.
- Judge a small difference with a bootstrap.
- Run trials and log facts
- Label each failure with a cause
- Count causes and slice by condition
- Form a hypothesis and fix one thing
- Re-test and compare with uncertainty
The one-sentence idea
Sort failures into causes, then fix the biggest.
Counts of causes, not a single percentage.
A success rate says how often, not why
“62% success” tells you nothing about what to fix. A policy can fail because it did not see the object, because it went to the wrong object, because it picked the right object clumsily, or because the hardware slipped. Each cause has a different remedy (more camera coverage, better instruction data, more demonstrations of the motion, a repaired gripper). So after measuring success (lesson 3.4), you sort the failures. The STAR-Gen taxonomy suggests one way to organise the tests by axis (visual, semantic and behavioural),[1] and lesson 8.4 develops failure types. This lesson is the practical routine: log, label, count, compare.
Step 1: log what happened, not just whether it worked
For each trial, record a few facts that can be checked afterwards (from video or sensors): was the target object visible, did the gripper close on the right object, did it close on anything, did the object end where it should? From these a small rule assigns a cause:
from collections import Counter
trials = [
{"target_visible": True, "went_to": "target", "grasped": True, "placed_ok": True},
{"target_visible": False, "went_to": None, "grasped": False, "placed_ok": False},
{"target_visible": True, "went_to": "other", "grasped": True, "placed_ok": False},
{"target_visible": True, "went_to": "target", "grasped": False, "placed_ok": False},
{"target_visible": True, "went_to": "target", "grasped": True, "placed_ok": False},
{"target_visible": True, "went_to": "other", "grasped": False, "placed_ok": False},
{"target_visible": True, "went_to": "target", "grasped": True, "placed_ok": True},
{"target_visible": False, "went_to": None, "grasped": False, "placed_ok": False},
]
def cause(t):
if t["placed_ok"]: return "success"
if not t["target_visible"]: return "perception: target not seen"
if t["went_to"] == "other": return "grounding: went to the wrong object"
if not t["grasped"]: return "action: failed to grasp"
return "action: placed wrongly"
counts = Counter(cause(t) for t in trials)
for name, n in counts.most_common():
print(f"{n} of {len(trials)} {name}")Output:
2 of 8 success 2 of 8 perception: target not seen 2 of 8 grounding: went to the wrong object 1 of 8 action: failed to grasp 1 of 8 action: placed wrongly
Toy model. Eight invented trials. A real analysis uses dozens or hundreds, and labels are best checked by two people watching the video.
Now the question has a shape: perception, grounding and action each show up. In a real study the biggest bar tells you where to spend effort first.
Step 2: slice by condition
Failures usually cluster. Break success down by the conditions you recorded (object, position, lighting) and the cluster shows:
import numpy as np
rng = np.random.default_rng(0)
# 300 invented trials: the policy is weak with the bottle and when the lights are dim
objects = rng.choice(["cup", "box", "bottle"], 300)
lights = rng.choice(["bright", "dim"], 300)
p = 0.9 - 0.4 * (objects == "bottle") - 0.3 * (lights == "dim")
success = rng.random(300) < p
for name, values in (("object", objects), ("light", lights)):
for v in np.unique(values):
m = values == v
print(f"{name:<7}{v:<8} success {success[m].mean():.2f} ({m.sum()} trials)")Output:
object bottle success 0.28 (115 trials) object box success 0.82 (95 trials) object cup success 0.73 (90 trials) light bright success 0.74 (141 trials) light dim success 0.45 (159 trials)
Always show the number of trials beside each rate (lesson 3.4): a rate from 5 trials is not much to go on. The bottle and dim light stand out here, so the next step is a hypothesis (the gripper cannot hold round shapes; the camera is noisy in low light) that you can test.
Step 3: is policy B really better than A?
After a fix you have two success counts. A difference may be luck. A bootstrap resamples the trials with replacement many times to see how often the observed ordering would flip:
import numpy as np
rng = np.random.default_rng(0)
a = np.array([1] * 31 + [0] * 19) # policy A: 31 successes in 50 trials
b = np.array([1] * 36 + [0] * 14) # policy B: 36 successes in 50 trials
wins = 0
resamples = 5000
for _ in range(resamples):
ra = rng.choice(a, size=len(a), replace=True).mean()
rb = rng.choice(b, size=len(b), replace=True).mean()
wins += rb > ra
print("A:", a.mean(), "B:", b.mean())
print("share of resamples in which B beats A:", wins / resamples)Output:
A: 0.62 B: 0.72 share of resamples in which B beats A: 0.8316
B looks 10 points better but is better in only about four out of five resamples: suggestive, not conclusive. More trials (or running both policies on the same scenes) would sharpen the answer. Resist writing “B improves success by 10%” from this.
Habits that make the analysis trustworthy
Decide the failure categories before looking at results, otherwise you will invent categories that fit what you saw. Fix the number of trials and the scenes in advance, and run the old and new policy on the same scenes so differences are not just different luck. Keep the videos: a label you cannot re-check is an opinion. Write down the version of the model, the data, the statistics file and the code for every run (lesson 7.2). And report what did not improve as well as what did; the next person, or you in three months, will need it.
Common misconceptions
Misconception 1 “A higher success rate means the fix worked.”
Correction With 50 trials, a 10-point gain flipped in about one in five resamples.
Misconception 2 “Failure is failure.”
Correction Perception, grounding and action failures need different remedies.
Misconception 3 “Aggregate results are enough.”
Correction Slices reveal clusters that averages hide.
Engineering takeaways
- Log facts.
- Label causes.
- Slice by condition.
- Bootstrap small differences.
Count why, not just how often.
Knowledge check
Try each question first, then reveal the explanation.
Question 1
A robot reaches for the wrong object. Which cause?
Explanation
Grounding.
Question 2
Why show trial counts next to rates?
Explanation
A rate from few trials is uncertain.
Question 3
What does a bootstrap estimate here?
Explanation
How often the observed ordering would flip with resampled trials.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2025). A taxonomy for evaluating generalist robot policies. arXiv; later versions retitled “…generalist robot manipulation policies” (IEEE RA-L). arXiv:2503.01238
Cited for:
- Proposes a taxonomy for generalisation in robot manipulation with three axes that follow the policy’s input and output modalities: visual (perturb the images), semantic (perturb the language) and behavioural (perturb the actions). Checked against the paper’s abstract.
What’s next
Next: simulation-based manipulation experiments.