5.2 — RT-2

Learning objectives

After this lesson, you should be able to:

• Explain co-fine-tuning on web and robot data.
• Explain how actions become text and why outputs must be checked.
• Explain, with the toy, how a similar word can borrow a skill.
• State the limit of RT-2’s generalisation.

The one-sentence idea

Teach a web-trained model to speak actions.

New words can transfer; new motions cannot.

The big idea: borrow the web’s knowledge

RT-1 learned only from robot data, so it knew only the objects and words that appeared in it. RT-2 asks: why not start from a vision-language model that already knows a huge amount about the world from Internet-scale images and text? The authors study how such models can be incorporated directly into end-to-end robotic control to boost generalisation and enable emergent semantic reasoning.

The method is co-fine-tuning: take existing vision-language models and fine-tune them on both robotic trajectory data and Internet-scale vision-language tasks such as visual question answering, with actions expressed as text tokens in the same way as natural-language tokens.^[1] The paper names this model class vision-language-action (VLA) models.^[1] Instances were built on the PaLI-X (55B) and PaLM-E (12B) models.^[1] For a guided tour of the paper itself, with its experiments and a critical review, see the RT-2 paper guide.

Actions as a string

Because actions are text, the model’s answer to “what should the robot do?” is a short string of integers. The announcement shows an example, “1 128 91 241 5 101 127 217”, whose first number says whether to continue or terminate the episode.^[2] The decoding side is the same as in lesson 1.2.

Since a language model can in principle write any text, a real system needs a safeguard that the answer is a valid action. One common way is to allow only the valid number tokens at each position when generating. Our illustration, with random scores, shows why it matters:

import numpy as np
rng = np.random.default_rng(0)

vocab_size = 32000                                      # a language model vocabulary
valid = np.arange(256)                                   # token ids that stand for the numbers 0..255 (our assumption)

def sample(logits, allowed=None):
    if allowed is not None:
        mask = np.full_like(logits, -np.inf); mask[allowed] = 0
        logits = logits + mask
    p = np.exp(logits - logits.max()); p /= p.sum()
    return rng.choice(len(logits), p=p)

trials = 2000
free, masked = 0, 0
for _ in range(trials):
    logits = rng.normal(0, 1, vocab_size)
    logits[valid] += 3.0                                  # the model has learned to prefer action tokens
    free   += sample(logits) >= 256
    masked += sample(logits, valid) >= 256
print("invalid action tokens without restriction:", free, "of", trials)
print("invalid action tokens with restriction:   ", masked, "of", trials)

invalid action tokens without restriction: 1711 of 2000
invalid action tokens with restriction:    0 of 2000

Toy model. Random scores with an assumed vocabulary layout. It shows a general risk of free-text outputs, not how RT-2 is implemented.

With 31,744 other tokens competing, a modest preference for the right ones still leaves most samples invalid in this toy (1,711 of 2,000). A trained model is far more peaked than random scores, so the real rate is much lower, but it is not zero in principle, and when the output moves a motor even rare invalid outputs are not acceptable.

How web knowledge can reach the robot

Suppose the robot data only ever showed “pick up the tiger toy”, but the web knowledge says a lion is a lot like a tiger. A model whose language and vision representations reflect that similarity can apply the skill to “pick up the lion toy” without a single lion demonstration. The toy below builds that in the simplest way, with hand-made word vectors:

import numpy as np

# "web knowledge": similar things have similar vectors
words = {"tiger": [0.9, 0.1, 0.0], "lion": [0.85, 0.15, 0.05], "cup": [0.0, 0.2, 0.9], "banana": [0.1, 0.9, 0.1]}
vec = {w: np.array(v) for w, v in words.items()}

# "robot data": grasp positions were demonstrated for these objects only
demonstrated = {"tiger": (0.2, 0.4), "cup": (-0.5, 0.1), "banana": (0.7, -0.3)}

def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def act(instruction_word):
    best = max(demonstrated, key=lambda w: cosine(vec[instruction_word], vec[w]))      # closest known object
    return best, demonstrated[best]

for word in ("tiger", "lion"):
    used, grasp = act(word)
    print(f"pick up the {word}: uses the skill learned for '{used}' -> grasp at {grasp}")

pick up the tiger: uses the skill learned for 'tiger' -> grasp at (0.2, 0.4)
pick up the lion: uses the skill learned for 'tiger' -> grasp at (0.2, 0.4)

Toy model. Hand-made vectors and a lookup. RT-2 does not work by lookup, but the principle, that shared representations let skills carry over to new words, is what the paper reports as better generalisation to novel objects.

The paper’s evaluation, about 6,000 trials, reports better generalisation to novel objects, following commands absent from the robot data (for example placing an object on a specific number or icon) and basic reasoning such as picking the smallest object or the one nearest another.^[1] Adding chain-of-thought reasoning enabled multi-step semantic reasoning, such as choosing an improvised tool.^[1]

The honest limit

The same paper states a limit: the model’s physical skills remain limited to the distribution of skills seen in the robot data, although it can apply them in new ways using web knowledge.^[1] It can be told a new what, but not taught a new how by the web alone. A skill that was never demonstrated (a new kind of motion) is still missing. This is the dividing line to remember between semantic generalisation and physical generalisation (lesson 5.5).

Common misconceptions

Engineering takeaways

1. Co-fine-tune on web and robot data.
2. Actions are text tokens.
3. Semantic generalisation improves.
4. Physical skills stay within the robot data.

The web taught it what things are; the robot data taught it how to move.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: Open X-Embodiment, pooling demonstrations from many robots.

Back to the VLA roadmap