1.2 — What Is a Vision-Language-Action Model?

Learning objectives

After this lesson, you should be able to:

• State the inputs and the output of a VLA.
• Explain how a continuous action is discretised into tokens and decoded again.
• Compute the precision limit set by the number of bins.
• Name two ways VLAs produce actions: as tokens or as a generated continuous chunk.

The one-sentence idea

A VLA turns what the robot sees and what it was told into the next robot action.

The model outputs actions, not a plan in words.

What the name means

A vision-language-action (VLA) model takes camera images and a language instruction as input and produces robot actions as output. The name was introduced with RT-2, whose authors used it for vision-language models that are co-fine-tuned on robot trajectories together with Internet-scale vision-language tasks.^[1] A survey of the field also credits RT-2 with introducing the term.^[2]

Each word in the name is an input or output of the model: vision (what the cameras see), language (what the person asked for) and action (what the robot should do next). The interface is therefore a function, and the next example writes it down as one. The model inside is a stub that returns zeros, because the point is the shape of the interface, not the answer.

import numpy as np

def vla(image, instruction, state):
    """image: height x width x 3 camera frame; instruction: text; state: robot joint readings.
    Returns the next action: 6 end-effector motion values plus a gripper command."""
    assert image.ndim == 3 and image.shape[2] == 3
    return np.zeros(7)

image = np.zeros((224, 224, 3), dtype=np.uint8)       # a blank frame stands in for the camera
action = vla(image, "pick up the red block", state=np.zeros(7))
print("input image shape:", image.shape)
print("output action shape:", action.shape)

input image shape: (224, 224, 3)
output action shape: (7,)

Toy model. The function body is a placeholder. The sizes (224 by 224 pixels, a 7-number action) are common choices, not a requirement of the definition.

Two things in the signature are worth noticing. The image is raw pixels, not a list of objects found by a separate detector, and the output is a command for the robot, not text describing a plan. Those two properties are what separate a VLA from the systems compared in lesson 1.3.

Why build one this way

The motivation in the RT-2 paper is transfer: a model that has already learned from Internet-scale images and text carries that knowledge into robot control. The authors do this by writing robot actions as text tokens so that the model handles them in the same way as language.^[1] The same paper is also clear about a limit: the physical skills of the model stay limited to the distribution of skills seen in the robot data, although it can apply those skills in new ways using the web knowledge.^[1] Language and vision knowledge generalises; motion skills still have to be demonstrated.

Models in this family share the idea but differ in how they produce actions. The rest of the lesson looks at the two main ways, and the architecture lesson 1.4 puts them in a pipeline.

Actions as tokens

A language model predicts a token from a fixed vocabulary. To reuse it for control, the continuous action has to become tokens. RT-1 discretises each action dimension into 256 bins.^[3] OpenVLA does the same: each action dimension is split into 256 bins between the 1st and 99th quantile of the training actions, the bins are written over the 256 least-used tokens of the Llama tokenizer, and the model is trained with next-token prediction on those action tokens.^[4] At run time the predicted tokens are decoded to continuous values and de-normalised to the robot’s physical scale.^[4]

The next example does the same thing in plain NumPy, with simpler bin limits (the range of the data, not quantiles). It encodes a 7-number action to 256-level integers and decodes it again:

import numpy as np

BINS = 256
low  = np.array([-0.05, -0.05, -0.05, -0.5, -0.5, -0.5, 0.0])    # per-dimension limits
high = np.array([ 0.05,  0.05,  0.05,  0.5,  0.5,  0.5, 1.0])

def encode(action):
    unit = (np.clip(action, low, high) - low) / (high - low)      # 0..1 in each dimension
    return np.minimum((unit * BINS).astype(int), BINS - 1)         # integer bin 0..255

def decode(tokens):
    unit = (tokens + 0.5) / BINS                                   # centre of the bin
    return low + unit * (high - low)

action = np.array([0.013, -0.021, 0.004, 0.10, -0.25, 0.0, 1.0])
tokens = encode(action)
print("tokens:", tokens.tolist())
print("decoded:", np.round(decode(tokens), 4).tolist())
print("largest error:", round(float(np.abs(decode(tokens) - action).max()), 5))

tokens: [161, 74, 138, 153, 64, 128, 255]
decoded: [0.0131, -0.0209, 0.0041, 0.0996, -0.248, 0.002, 0.998]
largest error: 0.00195

Toy model. The limits, the 7-number action layout and the centre-of-bin decoding are our own simple choices. OpenVLA uses quantile-based limits and maps bins onto language-model tokens; RT-1 and RT-2 differ in details too.

The decoded action is close to the original but not identical. Discretisation throws away detail, and the error depends on how many bins there are. For one dimension covering a range R with N bins, the worst case when decoding to the bin centre is R / (2N), arithmetic you can check with the next example:

for bins in (16, 64, 256):
    worst = 0.1 / (2 * bins)                       # a 0.1 m range, as in the position limits above
    print(bins, "bins -> worst-case error", round(worst * 1000, 3), "mm")

16 bins -> worst-case error 3.125 mm
64 bins -> worst-case error 0.781 mm
256 bins -> worst-case error 0.195 mm

This is why 256 bins is a sensible default for coarse manipulation, and also why it is a limit: fine motions need either a narrower range per step or a different output, such as the continuous action generators in the next section.

Actions written as text

RT-2’s announcement shows an action as a string of integers, for example “1 128 91 241 5 101 127 217”, where the first number says whether the episode should continue or terminate.^[5] That is literally text, so any language model can emit it. The example below parses a string in that style into an episode flag and a list of values. It shows only the mechanics of reading such output; the meaning of each position in a real model is defined by that model’s training, and we do not claim a mapping for the example string.

reply = "1 128 91 241 5 101 127 217"                # the example string shown in the RT-2 announcement
numbers = [int(t) for t in reply.split()]
flag, values = numbers[0], numbers[1:]
print("first number (continue / terminate flag):", flag)
print("remaining", len(values), "numbers:", values)
print("all inside 0..255:", all(0 <= v <= 255 for v in values))

first number (continue / terminate flag): 1
remaining 7 numbers: [128, 91, 241, 5, 101, 127, 217]
all inside 0..255: True

A model that emits text can also emit garbage text, so a real system must check that the output parses and falls in range before it is sent to a motor. That check is a robotics-engineering step, not part of the definition, but it is where a text interface meets a physical consequence (lesson 1.1).

Another way: generating the action directly

Not every VLA writes actions as language tokens. π0 builds on a pretrained vision-language model and adds a flow-matching architecture, so that it keeps the Internet-scale semantic knowledge.^[6] It pairs the vision-language model with a separate set of weights for the action outputs, called the action expert, which generates action chunks with flow matching.^[6] Its vision-language backbone is PaliGemma, a model of about 3 billion parameters made from a SigLIP image encoder and the Gemma-2B language model.^[7] How flow matching works is outside this lesson and is covered when we get to action heads; for now the point is that “VLA” names the input and output, not one particular way of producing the output.

Action chunking, producing several future actions at once, is used in other robot policies too, for instance ACT,^[8] and diffusion policy executes part of the predicted sequence before querying the policy again.^[9] Lesson 1.5 returns to why that matters.

Where the training data comes from

The robot part of the training data is a set of demonstrations. OpenVLA was trained on 970,000 real-world robot demonstrations,^[4] and the Open X-Embodiment collaboration pooled more than a million trajectories from 22 robot embodiments.^[10] In that work, a high-capacity model trained on the pooled data showed positive transfer, improving several robots using experience from other platforms.^[10] That is the rationale for “generalist” policies, and the evidence is from specific models and tasks, not a guarantee for every robot.

Common misconceptions

Engineering takeaways

1. A VLA maps images and an instruction to an action.
2. The term was introduced with RT-2.
3. Actions can be written as tokens, which are decoded, or generated as continuous chunks.
4. Robot skills still come from robot data.

A VLA is a model whose answer is a movement.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next, how a VLA differs from a language-model planner and from a plain vision-language model.

Back to the VLA roadmap