2.5 — How Visual and Language Representations Condition Robot Actions

Learning objectives

After this lesson, you should be able to:

• Explain what it means for an action to be conditioned on an image and an instruction.
• Show with an ablation why both inputs are needed.
• Describe a token output versus a chunk output.
• Explain where the conditioning comes from.

The one-sentence idea

The action depends on what the robot sees and on what it was told.

Remove either input and the robot cannot act correctly.

The question this lesson answers

Lessons 2.1 to 2.4 built a model that turns a picture and a sentence into a sequence of vectors. A robot needs one more step: what should I do next? The action must depend on both inputs. The picture says where things are. The instruction says which of them matters. Take either away and the robot cannot act correctly. This idea is called conditioning: the action is conditioned on the observation and on the instruction.

RT-2 shows that this can be done with the same machinery as language: it writes robot actions as text tokens so that the model handles them in the same way as language.^[1] The rest of this lesson makes the dependence concrete.

The instruction picks the target

Here is the smallest version. A table holds a red block and a blue block at random places. The “image” is the list of object positions; the instruction says which colour to fetch. The policy scores each object against the instruction (as attention does in lesson 2.1) and moves toward the winner:

import numpy as np

colours = {"red": np.array([1.0, 0.0]), "blue": np.array([0.0, 1.0])}     # a tiny "embedding" of each colour

def policy(objects, instruction, hand):
    """objects: {colour: position}. Returns a move for the hand."""
    q = colours[instruction]                                  # what the instruction is looking for
    names = list(objects)
    scores = np.array([q @ colours[n] for n in names]) * 5    # match each object against it
    w = np.exp(scores - scores.max()); w /= w.sum()           # softmax: who matches best
    target = sum(wi * objects[n] for wi, n in zip(w, names))  # a weighted average of positions
    return target - hand

hand = np.array([0.0, 0.0])
objects = {"red": np.array([0.8, 0.2]), "blue": np.array([-0.5, 0.6])}
for instruction in ("red", "blue"):
    print("pick the", instruction, "block -> move", np.round(policy(objects, instruction, hand), 2).tolist())

pick the red block -> move [0.79, 0.2]
pick the blue block -> move [-0.49, 0.6]

Toy model. Hand-made colour vectors and a one-step “policy”. A VLA learns this kind of dependence from demonstrations instead of having it written down.

Same scene, different sentence, different move. The language did not change where the blocks are; it changed which one the policy attends to.

Take away the image, or the words

Now remove each input in turn and see what breaks. “No image” means the policy does not know where the blocks are, so it aims at the middle of the table. “No instruction” means it picks a block at random. Over 1,000 random scenes:

import numpy as np
rng = np.random.default_rng(0)

def success(use_image, use_instruction, trials=1000, tol=0.2):
    wins = 0
    for _ in range(trials):
        red, blue = rng.uniform(-1, 1, 2), rng.uniform(-1, 1, 2)
        wanted = red if rng.random() < 0.5 else blue          # the instruction asks for one of them
        if use_image and use_instruction:
            aim = wanted
        elif use_image:                                        # sees both blocks, does not know which one
            aim = red if rng.random() < 0.5 else blue
        elif use_instruction:                                  # knows the colour, cannot see where it is
            aim = np.zeros(2)
        else:
            aim = np.zeros(2)
        wins += np.linalg.norm(aim - wanted) < tol
    return wins / trials

print("image + instruction:", success(True, True))
print("image only:         ", success(True, False))
print("instruction only:   ", success(False, True))
print("neither:            ", success(False, False))

image + instruction: 1.0
image only:          0.519
instruction only:    0.034
neither:             0.042

Toy model. A made-up task and made-up policies. The numbers show why conditioning needs both inputs; they do not describe any real robot.

Only the version that uses both succeeds reliably. Image only gets it right about half the time, because it is a coin flip between the two blocks. Without the image, aiming at the middle almost never hits (about 3 to 4 times in 100).

What comes out: a token, or a chunk

The fused vector from the backbone must become an action. There are two broad families, both met in lesson 1.2. In the token family, OpenVLA splits each action dimension into 256 bins, writes them as tokens and trains with next-token prediction.^[2] In the continuous family, π0 pairs a pretrained vision-language model with a separate set of weights for the action outputs, the action expert, which generates action chunks with flow matching.^[3] ACT is another policy that predicts a whole chunk of actions at once, and its paper uses action chunking and temporal ensembling to reduce compounding errors.^[4]

The simplest possible “action head” is one more matrix that maps the backbone’s vector to a chunk of future actions. Only the shapes matter here:

import numpy as np
rng = np.random.default_rng(0)

d_model, horizon, action_dim = 128, 10, 7
summary = rng.normal(size=d_model)                       # the backbone's vector for "what to do now"
W_head = rng.normal(size=(d_model, horizon * action_dim)) * 0.05
chunk = (summary @ W_head).reshape(horizon, action_dim)
print("chunk of actions:", chunk.shape, "= 10 future steps x 7 numbers")
executed = chunk[:4]                                     # execute part of it, then look again (lesson 1.5)
print("executed before re-observing:", executed.shape)

chunk of actions: (10, 7) = 10 future steps x 7 numbers
executed before re-observing: (4, 7)

Toy model. A random linear head to show the output shape. Real heads are trained, and π0’s action expert uses flow matching rather than a single matrix.

Executing only part of the chunk before looking again is the receding-horizon idea from lesson 1.5.

Where the conditioning is learned

None of this is hand-written in a real VLA. The dependence of action on image and instruction is learned from demonstrations, each a recording of a camera view, an instruction and the actions a person or controller took. That is why the data in lesson 1.2 matters, and why the next phase of the roadmap is about what is recorded and how a policy is trained from it. Pooled datasets such as Open X-Embodiment, with more than a million trajectories from 22 robot embodiments,^[5] are one way to get enough of it.

Common misconceptions

Engineering takeaways

1. Action = f(image, instruction).
2. Both inputs are needed.
3. Outputs are tokens or chunks of continuous actions.
4. The dependence is learned from demonstrations.

The picture says where; the words say which; the action says how.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the vision-language foundations. Next: how robot data is recorded and how a policy learns from it.

Back to the VLA roadmap