8.4 — Perception, Grounding and Action Failures

Learning objectives

After this lesson, you should be able to:

• Define perception, grounding and action failures.
• Find the bottleneck stage with an oracle experiment.
• Read a confusion table for systematic bias.
• Tally action failures by motion phase.

The one-sentence idea

Isolate the stage, then fix that stage.

Stages multiply: one weak stage caps everything.

Three places a VLA can go wrong

Following the pipeline of lesson 1.4, a task can fail at three stages. Perception: the model misread the scene (did not see the object, or saw it in the wrong place). Grounding: the model saw the scene but misunderstood which thing the instruction refers to (picked the wrong object). Action: the model knew the right target but moved wrongly (a clumsy grasp, a collision, a slip). Lesson 7.6 sorted failures by labelling videos; this lesson gives you tools to test which stage is at fault.

The three map onto the axes of STAR-Gen:^[1] perception errors show up under visual perturbations, grounding errors under semantic ones, and action errors under behavioural ones. The same study reports that current VLAs struggle with several kinds of semantic generalisation,^[1] so grounding is a place to look hard.

The oracle experiment

To find the weakest stage, replace one stage at a time with a perfect one (an oracle) and see how much the end result improves. The stage whose repair helps most is the bottleneck. In a toy pipeline where each stage has its own accuracy:

import numpy as np
rng = np.random.default_rng(0)

accuracy = {"perception": 0.90, "grounding": 0.80, "action": 0.70}       # how often each stage is right, on its own

def run(perfect=(), trials=20000):
    wins = 0
    for _ in range(trials):
        ok = all(stage in perfect or rng.random() < acc for stage, acc in accuracy.items())
        wins += ok
    return wins / trials

print("real pipeline:             ", round(run(), 3))
for stage in accuracy:
    print(f"with a perfect {stage:<10}:", round(run(perfect=(stage,)), 3))

real pipeline:              0.503
with a perfect perception: 0.568
with a perfect grounding : 0.627
with a perfect action    : 0.729

Toy model. Three independent stages with invented accuracies. In a real VLA the stages are not separable modules, so oracles must be approximated, for example by giving the policy the object’s true position, or telling it the target by an unambiguous label.

Fixing the weakest stage (action, at 0.70) improves the result most, and fixing perception gains least. Because the stages multiply, the end-to-end rate (about 0.5) is lower than any single stage's.

Grounding errors have patterns

When a policy goes to the wrong object, the way it is wrong is informative. A confusion table of instructed object versus chosen object exposes systematic bias, such as always preferring the leftmost object or confusing similar colours:

import numpy as np
rng = np.random.default_rng(0)

objects = ["red cup", "red box", "blue cup"]
counts = np.zeros((3, 3), dtype=int)                       # rows: instructed; columns: chosen

for _ in range(600):
    want = rng.integers(3)
    # invented behaviour: "red cup" and "red box" are confused with each other 30% of the time; "blue cup" is rarely confused
    if want == 0:   probs = [0.70, 0.25, 0.05]
    elif want == 1: probs = [0.30, 0.65, 0.05]
    else:           probs = [0.02, 0.03, 0.95]
    counts[want, rng.choice(3, p=probs)] += 1

print(f"{'instructed':<12}" + "".join(f"{o:>10}" for o in objects))
for i, o in enumerate(objects):
    print(f"{o:<12}" + "".join(f"{n:>10}" for n in counts[i]))

instructed     red cup   red box  blue cup
red cup            132        49         6
red box             62       139        15
blue cup             5         4       188

The two red objects are confused with each other, not with the blue one: the model understands colour but not the object type. The remedy is targeted: more data or instructions that contrast red cups with red boxes, not “more data” in general.

Action failures: right target, wrong motion

Action failures are the ones where the gripper went to the right place but the grasp, the force or the path failed. Typical signs: the policy hovers (indecision between modes, lesson 3.4), collides with something near the target, or closes the gripper too early. Recording where in the motion the failure occurs is more useful than a yes/no: during approach, grasp, lift or place. A tally by phase is a start:

from collections import Counter

failure_phase = ["approach", "grasp", "grasp", "lift", "grasp", "place", "grasp", "approach", "grasp", "lift"]
for phase, n in Counter(failure_phase).most_common():
    print(f"{phase:<9} {n} of {len(failure_phase)} failures")

grasp     5 of 10 failures
approach  2 of 10 failures
lift      2 of 10 failures
place     1 of 10 failures

If half of the failures (5 of 10 here) are at the grasp, look at the gripper, the contact data and the grasp demonstrations, not at the language or the vision.

Common misconceptions

Engineering takeaways

1. Perception, grounding, action.
2. Oracles reveal the bottleneck.
3. Confusion tables show bias.
4. Tally by phase.

Find the stage, then the fix.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: whether a policy can recover from mistakes and cope with disturbances.

Back to the VLA roadmap