Skip to the walkthrough
FixTheRobot Home / VLA / Paper guides / RT-2

VLA · Paper guide

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

The paper that turned a web-trained vision-language model into a robot policy by writing actions as text, explained through the questions an engineer would ask.

  • Brohan, Brown, Carbajal, Chebotar et al.
  • arXiv:2307.15818 (v1)
  • Overview: about 4 minutes
  • Full guide: about 50 minutes

This is a guide to the RT-2 paper, not a sentence-by-sentence paraphrase. It is the paper that introduced the name vision-language-action model, so it is the natural starting point before π0, GR00T N1 and AsyncVLA, which keep the idea of building on a vision-language model but generate actions differently. Part A is a 4-minute overview. Parts B to D explain the method, the data and the experiments one idea at a time. Equations and details are folded away in Part E, and Part F is a critical review of the paper’s numbers and claims.

Start with the overview Jump to the critical review

How to use this page

  1. In a hurry: read Part A (about 4 minutes).
  2. To understand the method: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example, then technical details and where to find them in the paper.
  3. To check or rebuild it: open the folded sections (the action format, the objective, the baselines, runnable examples) and read Part F.

Open the paper (arXiv:2307.15818) beside this page; section, table and figure references are to it. We paraphrase and never reproduce its text. Statements are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own small model of an idea, and are never presented as results of the paper.

Part A · about 4 minutes

Start here: the paper at a glance

The problem

A robot that learns only from its own demonstrations knows only the objects, places and words that appear in them. Models trained on huge collections of Internet images and text know far more, but they answer in words, while a robot needs precise hand movements. Earlier work mostly used such models as a planner at the top, which turns a request into steps that a separate low-level controller carries out; that controller never benefits from the web knowledge. This paper asks: can a large pretrained vision-language model (VLM), a model that reads images and text and writes text, be used directly for low-level robot control?

The main contribution

  1. A model class: vision-language-action (VLA) models. A VLA is a vision-language model that writes robot actions as text tokens, in the same way as it writes words, with all of its weights shared between language and action tasks.[1]
  2. A training recipe: co-fine-tuning. Keep training the pretrained model on robot demonstrations and on the original web question-answering data at the same time, so it does not forget the web concepts.[1]
  3. Two instances, called RT-2. RT-2-PaLI-X (5 billion and 55 billion parameters) and RT-2-PaLM-E (12 billion), plus a way of serving them from the cloud fast enough for closed-loop control.[1]
  4. An evaluation of about 6,000 real-robot trials of generalisation, of “emergent” abilities that come from web knowledge, of design choices and of a chain-of-thought variant.[1]

How the model works, in one paragraph

At every step the robot’s camera image and the instruction are given to the model as a question, such as “what action should the robot take to put the apple in the bowl?”. The model answers with eight whole numbers, each written as a text token: one that says whether to stop, six that give how far to move and rotate the hand, and one for the gripper.[1] Each movement number is one of 256 slots between a minimum and a maximum. The numbers are converted back into a movement, the robot moves, the camera takes a new picture, and the loop repeats. While being trained, the same model also answers ordinary web questions about images, which is what keeps its general knowledge.

What it learns from, and what it is tested on

Learns from: web-scale image-and-text data (a filtered set of about 1 billion examples plus many captioning and question-answering sets), mixed with the robot demonstrations of the earlier RT-1 system: 13 robots, 17 months, an office kitchen. Robot data makes up about half (PaLI-X) or two thirds (PaLM-E) of the training mixture. Tested on: a real mobile manipulator with an arm, on tasks it has seen, on new objects, backgrounds and environments, and on instructions that need symbols, simple reasoning or recognising people, plus a simulated pushing benchmark called Language-Table.

What the paper reports

Selected values from Tables 1, 4, 5 and 6, as printed. Each baseline is trained on the same robot data.
MeasureRT-2Best baseline
Tasks seen in training (success, %)91 (PaLI-X-55B), 93 (PaLM-E-12B)92 (RT-1)
New objects, backgrounds, environments (average, %)62 and 6235 (MOO), 32 (RT-1)
Emergent instructions (average, %)60 (PaLI-X-55B), 40 (PaLM-E-12B)17 (RT-1)
Language-Table simulation (%)90 ± 10 (3-billion-parameter model)77 ± 4 (LAVA)
Training strategy, 55B model, unseen average (%)63 co-fine-tuned52 fine-tuned on robot data only

What to keep in mind

  • The gain is in generalisation. On tasks seen in training, RT-2 is level with the much smaller RT-1 (35 million parameters).
  • The robot does not learn new motions from web data; it uses the skills in its robot data in new ways.[1]
  • All robot results come from one lab, one kind of robot and trial counts that are small per condition, with no repeated runs or error bars; a few numbers in the tables disagree with each other (Part F).
  • The chain-of-thought result is shown only through example rollouts, not measured.

If you read nothing else: RT-2 turns a pretrained vision-language model into a robot policy by writing actions as numbers in text and training on robot and web data together; it matches the previous robot-only model on familiar tasks and does about twice as well on unfamiliar ones, but it needs a large model, runs at a few actions per second and learns no new motions.

Check your understanding: Why does RT-2 keep the web data during fine-tuning instead of training on robot data alone?

So that the model keeps seeing images and language from the web while it learns actions. The paper reports that this co-fine-tuning generalises better than fine-tuning on robot data alone, which it attributes to the model not forgetting its earlier concepts.

Part B

Understand the method

RT-2 adds almost nothing to an existing vision-language model: a convention for writing actions as numbers, a rule that limits what the model may write, and a way to serve it. Each idea below follows the same pattern, from intuition to where to find it in the paper.

How RT-2 turns an image and an instruction into a robot movement A camera image and an instruction are written as a question. A vision encoder turns the image into tokens. A large pretrained language model reads the image tokens and the text and writes eight number tokens, where only valid action tokens are allowed. The numbers are converted back into a stop flag, a movement and rotation of the hand and a gripper setting. The robot moves, the camera takes a new picture and the loop repeats. During training the same model also answers web questions about images. 1. Camera image + instructionwritten as a question: what action shouldthe robot take to [task]? 2. Vision encoderturns the image into a sequence of tokensPaLI-X: 22 billion parameters; PaLM-E: 4 billion 3. Large pretrained language modelreads image tokens and text togetherand writes its answer one token at a time 4. Eight number tokensonly valid action tokens are allowed herefor example: 1 143 129 123 145 114 115 127 5. Convert the numbers backstop flag, movement and rotation of therobot hand, and the gripper setting 6. The robot moves; the camera looks againrepeat at every control step55B model: 1 to 3 times a second closed loop: next image, next action During training, the same model alsoanswers web questions about imageswith ordinary text, so one set of weightsis shared between language and action
One step of control. Orange marks the part that is fine-tuned. Drawn by us from Sections 3.1 to 3.3 and Figure 1 of the paper; the example numbers are one of the paper’s printed outputs.

1. A vision-language model that answers with actions

Intuition
Teach a model that already understands pictures and words to answer “what should the hand do next?” the same way it answers “what is in this picture?”.
The problem it solves
Robot data is small and narrow, so a policy trained only on it knows few objects and words. Web-trained models know a lot but write text, and when they were used in robotics they mostly acted as planners above a separate controller that did not benefit from their knowledge.
How it works
Take a pretrained vision-language model, a model that reads one or more images plus text and writes text. Fine-tune it so that, when asked a robot question, it writes an action; when asked a web question, it still writes language. Nothing is added to the network for actions: the action is just another piece of text, so all weights are shared between the two kinds of task. The paper contrasts this with earlier systems (CLIPort, MOO) that combine a vision-language model with a policy of a specific structure, such as a restricted 2D action space or a calibrated camera.
Example
Illustration, not from the paper Image of a cluttered table, instruction “pick up the smallest object”. The model has seen many web images in which objects are described by size, so it can pick out the right one; the robot data teaches it what a pick looks like.
Why the authors chose it
It reuses a model that already cost a great deal to train, and it lets knowledge of objects, relations and words carry over to the robot.
Technical details
The paper restricts its definition to vision-language models of the form image and text in, text out, not to the embedding models such as CLIP that map images and text into a shared space. It builds on the data protocol of the earlier RT-1 system and names the model RT-2 because it replaces RT-1’s small network with a large vision-language model.
Evidence in the paper
§1 · §2 “Vision-language models” · §3.1 · Figure 1
Check your understanding: What is the difference between using a vision-language model as a planner and as a VLA?

A planner writes steps in words and a separate controller, which never saw web data, carries them out. In a VLA the same model writes the low-level actions, so web knowledge reaches the movements directly.

2. Actions as text: eight numbers

Intuition
A movement of the hand becomes a short sentence of eight numbers that the language model writes like any other words.
The problem it solves
A language model can only write tokens from its vocabulary, but a robot needs a continuous movement command.
How it works
The action is the stop command, the movement of the hand in three directions, its rotation in three directions and how far the gripper is open. Every dimension except the stop command is cut into 256 equal slots (bins), so the action is eight whole numbers. They are written in a fixed order, separated by spaces, as the target of the question “what action should the robot take to [task]?”. For PaLI-X, whose vocabulary has a separate token for each whole number up to 1,000, bin number n is written as the token for n. For PaLM-E, which has no such tokens, the 256 least-used tokens are overwritten and used for the bins.
Example
Illustration, not from the paper A small movement to the left with a slightly closed gripper might be written as eight numbers, most of them close to the middle value, since the middle bins mean “almost no movement”.
Why the authors chose it
Reusing the language model’s own output makes the action part of the same next-word prediction as every other task, with nothing new to train from scratch.
Technical details
The encoding follows RT-1. The paper does not print the physical range of each dimension, so the example below uses an invented range from −1 to +1 and shows the idea of equal slices and of rounding to the middle of a slice.
Run the example: a continuous action becomes text tokens
Toy illustration

What each part does

  1. Cut an invented range −1 to +1 into 256 equal slices.
  2. Convert one made-up action to bin numbers and write it as a string.
  3. Decode it again and measure the largest rounding error over many random values.
import numpy as np

# How a continuous action becomes text. The paper uses 256 equal-width bins per continuous dimension.
# It does not print the physical range of each dimension, so we use a made-up range of -1 to +1.
BINS = 256
LOW, HIGH = -1.0, 1.0
width = (HIGH - LOW) / BINS

def to_bin(value):
    # which of the 256 equal slices of [LOW, HIGH] the value falls in (0 to 255)
    return int(min(BINS - 1, max(0, np.floor((value - LOW) / width))))

def from_bin(b):
    # decode to the middle of the slice
    return LOW + (b + 0.5) * width

# one made-up action: terminate flag, then x y z move, x y z rotation, gripper
terminate = 1
values = [0.004, -0.62, 0.31, -0.99, 0.12, 0.0, 0.5]
tokens = [terminate] + [to_bin(v) for v in values]
print("tokens as the model would write them:", " ".join(str(t) for t in tokens))
print("decoded values:", [round(from_bin(b), 3) for b in tokens[1:]])

# the worst possible rounding error is half a slice
rng = np.random.default_rng(0)
many = rng.uniform(LOW, HIGH, 100_000)
error = np.abs(np.array([from_bin(to_bin(v)) for v in many]) - many)
print(f"slice width {width:.5f}; largest error over 100,000 random values {error.max():.5f} (half a slice = {width / 2:.5f})")
print("number of distinct action vectors: 256 ** 7 =", f"{256 ** 7:.3e}")

Output:

tokens as the model would write them: 1 128 48 167 1 143 128 192
decoded values: [0.004, -0.621, 0.309, -0.988, 0.121, 0.004, 0.504]
slice width 0.00781; largest error over 100,000 random values 0.00391 (half a slice = 0.00391)
number of distinct action vectors: 256 ** 7 = 7.206e+16

The ranges are made up; the paper’s ranges come from RT-1.

Evidence in the paper
§3.2 · Figure 1
Check your understanding: What is the largest error that using 256 equal bins can cause, in terms of the range of one dimension?

Half of one bin, which is the range divided by 512, because each value is decoded to the middle of its slice.

3. The two language-model backbones: PaLI-X and PaLM-E

Intuition
Two existing models are reused unchanged in structure; RT-2 is what they become after the fine-tuning.
The problem it solves
The idea has to work on models that were designed differently, so the authors test it on two.
How it works
PaLI-X reads images with a 22-billion-parameter vision transformer (ViT), a network that cuts an image into patches, passes it through a projection layer and gives the tokens to an encoder-decoder language model of 32 billion parameters and 50 layers, which writes the answer token by token. PaLM-E is a decoder-only language model that turns images and other robot inputs into vectors in its own token space; the 12-billion-parameter model uses a 4-billion-parameter vision transformer. The paper trains PaLI-X at 5 billion and 55 billion parameters and PaLM-E at 12 billion, and a smaller PaLI model of 3 billion parameters for the simulated Language-Table benchmark.
Example
Illustration, not from the paper The same question and picture go into either model; PaLI-X encodes them and then decodes the answer, while PaLM-E treats the image vectors as extra words in one long sequence.
Why the authors chose it
Showing the recipe on two designs and several sizes makes it less likely that the result depends on one architecture.
Technical details
The listed parts of the 55-billion model add up to 54 billion, as expected. For the model called 3 billion parameters, the listed parts add up to 5 billion; the example checks both, and Part F comes back to this.
Run the example: do the model sizes add up?
Our calculation

What each part does

  1. Add the listed vision and language parts of PaLI-X and of the Language-Table PaLI model.
  2. Compare each sum with the size in the model’s name.
  3. Work out the time per action implied by the reported control rates.
# Model sizes and speed, as stated in the paper (Appendix D and Section 3.3).
# Parameters in billions.
print("Do the listed components add up to the model's name?")
components = {
    "PaLI-X (called 55B): ViT-22B + encoder-decoder 32B": (22 + 32, 55),
    "PaLI (called 3B):    ViT-G/14 2B + UL2-3B":         (2 + 3, 3),
}
for name, (total, called) in components.items():
    print(f"  {name:55s} sum = {total:3d}B   named {called}B")

print()
print("Control rate against model size (55B: 1 to 3 Hz, 5B: about 5 Hz)")
for name, low, high in (("RT-2-PaLI-X-55B", 1, 3), ("RT-2-PaLI-X-5B", 5, 5)):
    ms_slow, ms_fast = 1000 / high, 1000 / low
    tokens_per_second = (8 * low, 8 * high)
    print(f"  {name:16s} time per action {ms_slow:6.0f} to {ms_fast:5.0f} ms;"
          f" at least {tokens_per_second[0]} to {tokens_per_second[1]} output tokens per second (8 per action)")

print()
print(f"55B has {55 / 5:.0f} times the parameters of 5B but runs only {5 / 3:.1f} to {5 / 1:.1f} times slower")
print("the time per action includes the network trip to the cloud service; the paper does not split it")

Output:

Do the listed components add up to the model's name?
  PaLI-X (called 55B): ViT-22B + encoder-decoder 32B      sum =  54B   named 55B
  PaLI (called 3B):    ViT-G/14 2B + UL2-3B               sum =   5B   named 3B

Control rate against model size (55B: 1 to 3 Hz, 5B: about 5 Hz)
  RT-2-PaLI-X-55B  time per action    333 to  1000 ms; at least 8 to 24 output tokens per second (8 per action)
  RT-2-PaLI-X-5B   time per action    200 to   200 ms; at least 40 to 40 output tokens per second (8 per action)

55B has 11 times the parameters of 5B but runs only 1.7 to 5.0 times slower
the time per action includes the network trip to the cloud service; the paper does not split it

Arithmetic on numbers printed in Section 3.3 and Appendix D.

Evidence in the paper
§3.1 · §4 · Appendix D
Check your understanding: Why does RT-2 need two different overwritten-token strategies for PaLI-X and PaLM-E?

They tokenise numbers differently. PaLI-X already has one token for each whole number up to 1,000, so the bins reuse those; PaLM-E has no such tokens, so the least-used tokens are repurposed.

4. Output constraint: only valid action tokens

Intuition
When the robot is being controlled, the model may only write numbers that the robot can understand.
The problem it solves
A language model can write any text, so left alone it could write a word where an action number must be, and the robot could not execute it.
How it works
When the model is prompted with a robot-action question, decoding is limited to the tokens that stand for valid action bins; for ordinary vision-language questions, the full vocabulary stays available.
Example
Illustration, not from the paper At each of the eight positions the model chooses among 256 allowed number tokens (the stop command has its own small set), not among all tens of thousands of words.
Why the authors chose it
It guarantees that every output can be decoded into an action, whatever the model would otherwise have preferred.
Technical details
The paper does not say how the stop command is restricted, or what the model does if no valid token is likely. Lesson 5.2 shows with an illustration why such a rule is needed.
Evidence in the paper
§3.2 “Output Constraint”
Check your understanding: Does the constraint change what the model knows?

No. It only changes which tokens may be chosen when decoding a robot answer; the network’s weights are the same, and web questions are answered without the constraint.

5. Running a very large model in a control loop

Intuition
The model is too big for the robot, so the robot asks a cloud service for each action.
The problem it solves
Models of tens of billions of parameters cannot run on the computers normally carried by a robot.
How it works
The models are deployed in a service on several tensor processing units (TPUs, Google’s AI chips) in the cloud, which the robot queries over the network; one service can serve several robots. The 55-billion-parameter model runs at 1 to 3 Hz (times per second) and the 5-billion-parameter model at about 5 Hz. The authors state that, as far as they know, this is the largest model used for direct closed-loop robot control by more than an order of magnitude.
Example
Illustration, not from the paper A robot sends its latest image and instruction, receives eight numbers, moves for a fraction of a second, and asks again.
Why the authors chose it
It makes a 55-billion-parameter policy usable on a real robot without on-board hardware that does not exist yet.
Technical details
The reported rates include the network trip, which the paper does not separate from the model’s own time. At 1 to 3 Hz each action takes between a third of a second and one second, which suits the pick and place tasks tested but is slow for tasks that need quick reactions; the authors list inference speed as a limitation and suggest quantisation and distillation to speed it up.
Evidence in the paper
§3.3 · §5
Check your understanding: What limits the tasks RT-2 can be used for, other than the robot data?

The control rate of 1 to 3 Hz for the largest model, and the need for a connection to a cloud service, which together rule out tasks that need fast reactions.

Part C

Understand the data and the training

The training has two ingredients, web data and robot data, mixed in every batch. Everything else follows the settings of the original PaLI-X and PaLM-E papers.

The data: web question answering and robot demonstrations

Intuition
The model reads from two libraries at once: one that knows the world and one that knows this robot.
The problem it solves
The robot data covers a few kinds of objects and instructions; the web data covers almost everything but has no actions.
How it works
The web part is based on the data mixtures of PaLI-X and PaLM-E: mostly WebLI, about 10 billion image and text pairs in 109 languages, filtered to the 10% with the best image and text match, which gives 1 billion examples, plus captioning and question-answering sets. RT-2-PaLI-X does not use the “episodic” WebLI set. The robot part is the data of RT-1: demonstrations collected with 13 robots over 17 months in an office kitchen, each described by an instruction made of a verb for the skill (such as pick, open, place into) and nouns for the objects. For the simulated Language-Table benchmark, the model is trained on five tasks: predict the action from two frames and an instruction, and, to learn more about the scene, predict the instruction, the arm position, the number of steps between frames and whether the task succeeded.
Example
Illustration, not from the paper An image of a street with the question “what is happening?” and the answer “a donkey walks down the street”, next to an image of the robot’s table with the question “what action should the robot take to pick the can?” and eight numbers.
Why the authors chose it
The web part supplies concepts, the robot part supplies the mapping from concepts to motions.
Technical details
Section 4 and Appendix B. The paper gives no count of robot demonstrations; it points to the RT-1 paper.
PartWhat it isWhere it comes from
Webabout 1 billion filtered image–text examples, plus captioning and question answeringPaLI-X and PaLM-E training data
Robotdemonstrations of 13 robots over 17 months, one office kitchenRT-1 data
Language-Tablefive tasks on the simulated pushing benchmarkLanguage-Table data
Evidence in the paper
§4 · Appendix B
Check your understanding: What does the web data contribute that the robot data cannot?

Knowledge of many objects, words, symbols and relations that never appear in the robot demonstrations.

Co-fine-tuning: keep the web data in the mix

Intuition
Keep revising the old subject while learning the new one, so the old one is not forgotten.
The problem it solves
Fine-tuning on robot data only can make the model drift towards the narrow robot data and lose the broad concepts it learned from the web.
How it works
Every training batch mixes web examples with robot examples. To make sure the robot data is not drowned out, the sampling weight on the robot dataset is increased: it makes up about half of the mixture for RT-2-PaLI-X and about two thirds for RT-2-PaLM-E.
Example
Illustration, not from the paper A batch of 2,048 examples for the 55B model contains roughly a thousand robot examples and a thousand web question-answering examples.
Why the authors chose it
The authors argue that seeing both abstract visual concepts and low-level actions during fine-tuning gives more generalisable policies; they attribute the ablation result to the model not forgetting its earlier concepts.
Technical details

Our calculation Co-fine-tuning does not replay the web data in full. By our arithmetic, the 55B model sees about 82 million web examples in its 80,000 steps, under a tenth of the one billion filtered WebLI examples alone; the 5B model sees about 277 million. The example makes the calculation and assumes one example per batch element, which the paper does not say.

Run the example: how much web data co-fine-tuning sees
Our calculation

What each part does

  1. Multiply the number of steps by the batch size for each model.
  2. Take the web share of the mixture from the robot share.
  3. Compare with the one billion filtered WebLI examples.
# How much web data does co-fine-tuning actually see? (Appendix B and E)
# Examples seen = gradient steps x batch size (assuming one example per batch element).
runs = {
    #                    steps,    batch, robot share of the mixture
    "RT-2-PaLI-X-55B": (80_000,   2048, 0.50),
    "RT-2-PaLI-X-5B":  (270_000,  2048, 0.50),
    "RT-2-PaLM-E-12B": (1_000_000, 512, 0.66),
}
WEBLI = 1_000_000_000   # filtered WebLI examples; other web datasets are in the mix as well

print(f"{'model':16s} | examples seen | robot share | web examples | web examples / 1 billion")
for name, (steps, batch, robot_share) in runs.items():
    seen = steps * batch
    web = seen * (1 - robot_share)
    print(f"{name:16s} | {seen / 1e6:10.1f} M | {robot_share:10.0%} | {web / 1e6:9.1f} M | {web / WEBLI:6.2f}")

print()
print("ratio of training examples, 5B to 55B:", round(270_000 * 2048 / (80_000 * 2048), 2))
print("the 5B and the 55B model also differ in how long they were trained, not only in size")

Output:

model            | examples seen | robot share | web examples | web examples / 1 billion
RT-2-PaLI-X-55B  |      163.8 M |        50% |      81.9 M |   0.08
RT-2-PaLI-X-5B   |      553.0 M |        50% |     276.5 M |   0.28
RT-2-PaLM-E-12B  |      512.0 M |        66% |     174.1 M |   0.17

ratio of training examples, 5B to 55B: 3.38
the 5B and the 55B model also differ in how long they were trained, not only in size

Arithmetic on Appendix B and E; the assumption of one example per batch element is ours.

Evidence in the paper
§3.2 “Co-Fine-Tuning” · Appendix B · Appendix E · Figure 6b
Check your understanding: What does the 50% to 66% robot share tell you about the sampling weights?

The robot data, which is far smaller than the web data, is sampled much more often than its size alone would suggest; otherwise it would be a tiny part of every batch.

Training settings

Intuition
The loss is the usual next-word loss; RT-2 changes the data, not the optimiser.
The problem it solves
Large models are expensive to tune, so the authors avoid new tuning choices.
How it works
The models are trained with next-token prediction, which for robot examples is the same as the behaviour-cloning loss: predict the demonstrator’s action. The hyperparameters, including learning-rate schedules and regularisation, come from the original PaLI-X and PaLM-E papers. The main settings are in the table.
Example
Illustration, not from the paper For a robot example the model is scored on how well it predicts each of the eight number tokens, in order, given the image, the instruction and the numbers before it.
Why the authors chose it
It keeps the comparison simple: the only thing that differs from the original models is what they are trained on.
Technical details
Appendix E, as printed. The 5B and 55B models were trained for different numbers of steps.
ModelLearning rateBatch sizeGradient steps
RT-2-PaLI-X-55B1e-32,04880,000
RT-2-PaLI-X-5B1e-32,048270,000
RT-2-PaLM-E-12B4e-45121,000,000
RT-2-PaLI-3B (Language-Table)1e-3128300,000
Evidence in the paper
§4 · Appendix E
Check your understanding: Why is next-token prediction the same as behaviour cloning here?

The tokens of the answer are the demonstrator’s action, so predicting them correctly is imitating the demonstration.

Chain of thought: write the plan before the action

Intuition
Let the model say in words what it is about to do, then do it.
The problem it solves
Complicated instructions (“bring me something to hammer a nail”) need a step of reasoning before a movement.
How it works
The authors add a “Plan” step to the training data: the instruction, then a short sentence stating the purpose of the next action, then the action numbers. They fine-tune a variant of RT-2-PaLM-E on this for only a few hundred gradient steps. At run time the model writes the plan first and the action after.
Example
Illustration, not from the paper Instruction: “I am hungry.” Plan: “pick the chocolate bar.” Action: eight numbers.
Why the authors chose it
The plan connects the question-answering data, where reasoning happens in words, with the manipulation data, where actions are written.
Technical details
The paper presents example rollouts (Figures 7 and 10) and does not report success rates, a comparison with the model without the plan step or the number of trials.
Evidence in the paper
§4.4 · Figures 7 and 10
Check your understanding: What kind of evidence supports the chain-of-thought result?

Qualitative rollouts, a handful of examples. No success rate or baseline comparison is reported.

Part D

Understand the experiments

Every result comes from real-robot trials, with a simulated benchmark as an extra. This part explains how the tests were set up, what each table says and what it can and cannot show. Our re-calculations are in the folded examples, each marked as a calculation or a toy.

The set-up: robot, baselines and trials

How it works
Unless stated otherwise the robot is a mobile manipulator with seven degrees of freedom, controlled with the action format of Part B. Every baseline is trained on exactly the same robot data. RT-1 is a 35-million-parameter transformer without web pretraining. VC-1 and R3M are pretrained visual representations (R3M was learned from videos of human activity), feeding an RT-1-style network together with a language embedding; their weights are trained along with the policy. MOO uses a vision-language model as a separate module that marks the object of interest in the image with a coloured pixel, and a policy then uses the marked image. The tasks seen in training number over 200: 36 for picking, 35 for knocking over, 35 for placing upright, 48 for moving, 18 for drawers and 36 for picking from and placing into drawers. Unseen tasks are split into new objects, new backgrounds and new environments, each in an easy and a hard version, and the paper says they total over 280. Emergent abilities are tested with their own list of instructions, run five times each, and the models are compared one after another in the same conditions (an “A/B testing framework”).
Technical details

Our calculation Adding up the paper’s own lists gives 208 seen tasks, matching “over 200”, but only 138 instructions in Table 2 for the unseen scenarios, not over 280; the emergent list has 38 instructions, or 190 trials per model. At about two runs per task the totals reach the reported 6,000 trials. The paper does not say how many times each unseen instruction was run, beyond “between 1 and 5”.

Run the example: the number of tasks and trials
Our calculation

What each part does

  1. Add up the task groups listed for seen tasks and the instructions listed for unseen scenarios.
  2. Work out the number of emergent trials per model.
  3. Find the average number of runs per task that would add up to 6,000 trials.
# How many tasks and trials does the evaluation contain? The counts are ours, made from the paper's lists of instructions.
seen = {"pick objects": 36, "knock objects": 35, "place upright": 35, "move objects": 48,
        "open or close drawers": 18, "pick out of or place into drawers": 36}
unseen = {   # Table 2: number of instructions listed for each scenario
    "objects easy": 10, "objects hard": 21,
    "backgrounds easy": 12, "backgrounds hard": 17,
    "environments easy": 45, "environments hard": 33,
}
emergent_instructions = 3 + 4 + 3 + 4 + 5 + 5 + 8 + 3 + 3   # Table 3; each instruction was run 5 times

print("seen tasks:", sum(seen.values()), '(the text says "over 200")')
print("unseen instructions listed in Table 2:", sum(unseen.values()), '(the text says "over 280 tasks")')
print("emergent instructions:", emergent_instructions, "-> trials per model:", 5 * emergent_instructions)

# What average number of runs per task would add up to the reported "about 6,000" trials?
# Seen and unseen tasks were evaluated with six models; Table 6 adds four more models on the unseen tasks
# (its 55B co-fine-tuned row is the same model as in Table 4); four models were compared on the emergent tasks.
per_run = (sum(seen.values()) + sum(unseen.values())) * 6 + 4 * sum(unseen.values())
emergent_trials = 4 * 5 * emergent_instructions
print()
print("trials for one run of every task:", per_run, "| emergent trials:", emergent_trials)
runs = (6000 - emergent_trials) / per_run
print(f"average runs per task needed to reach 6,000 in total: {runs:.2f}")
print("the paper says each instruction was run between 1 and 5 times, so about 6,000 is plausible but cannot be checked")

Output:

seen tasks: 208 (the text says "over 200")
unseen instructions listed in Table 2: 138 (the text says "over 280 tasks")
emergent instructions: 38 -> trials per model: 190

trials for one run of every task: 2628 | emergent trials: 760
average runs per task needed to reach 6,000 in total: 1.99
the paper says each instruction was run between 1 and 5 times, so about 6,000 is plausible but cannot be checked

The counts are ours, made from the paper’s lists of instructions.

Evidence in the paper
§4 “Baselines” · §4.1 · Appendix C · Appendix F.2 · Tables 2 and 3
Check your understanding: Why is it useful that all baselines use the same robot data?

Any difference in success can then be attributed to the model and its pretraining rather than to one method having seen more demonstrations.

Seen tasks and generalisation (Table 4)

How it works
The headline test is how well a policy copes with things it has not seen. RT-2 matches RT-1 on seen tasks and is about twice as good as RT-1 and MOO on the unseen average, and about six times better than R3M and VC-1. The authors write that the PaLM-E version looks better on the harder scenarios and worse on the easier ones, giving a similar average to the PaLI-X version.
Technical details
Table 4, success rate in percent, as printed.
ModelSeenObjects easyObjects hardBackgrounds easyBackgrounds hardEnvironments easyEnvironments hardUnseen average
R3M4532141390212
VC-16334101330010
RT-1923143719261432
MOO755848384119335
RT-2-PaLI-X-55B9170629648633562
RT-2-PaLM-E-12B9384767571363362

Our calculation The unseen average is the plain mean of the six scenario columns, and it agrees with every printed average within rounding. The claimed pattern for PaLM-E holds only partly (Part F). The same example recomputes the ratios.

Run the example: re-check Table 4
Our calculation

What each part does

  1. Type in Table 4.
  2. Recompute each model’s unseen average and compare with the printed one.
  3. Rank the baselines, compute the ratios and compare PaLM-E with PaLI-X scenario by scenario.
# Table 4 (overall results), typed in as printed. Success rates in percent.
# Columns: seen tasks, then six unseen scenarios (objects easy/hard, backgrounds easy/hard, environments easy/hard), printed average.
rows = {
    "R3M":              (45, [32, 14, 13,  9,  0,  2], 12),
    "VC-1":             (63, [34, 10, 13,  3,  0,  0], 10),
    "RT-1":             (92, [31, 43, 71,  9, 26, 14], 32),
    "MOO":              (75, [58, 48, 38, 41, 19,  3], 35),
    "RT-2-PaLI-X-55B":  (91, [70, 62, 96, 48, 63, 35], 62),
    "RT-2-PaLM-E-12B":  (93, [84, 76, 75, 71, 36, 33], 62),
}
print("model              | mean of six unseen | printed")
for name, (seen, six, printed) in rows.items():
    mean = sum(six) / 6
    print(f"{name:18s} | {mean:18.2f} | {printed:7d} | {'ok' if abs(mean - printed) <= 0.5 else 'DIFFERENT'}")

print()
print("ranking of the four baselines by unseen average:",
      sorted(((rows[n][2], n) for n in ("R3M", "VC-1", "RT-1", "MOO")), reverse=True))
rt2 = rows["RT-2-PaLI-X-55B"][2]
for n in ("RT-1", "MOO", "R3M", "VC-1"):
    print(f"RT-2-PaLI-X-55B / {n:4s} = {rt2 / rows[n][2]:.1f}x")

print()
print("PaLM-E minus PaLI-X, by scenario (points):")
names = ["objects easy", "objects hard", "backgrounds easy", "backgrounds hard", "environments easy", "environments hard"]
pali, palm = rows["RT-2-PaLI-X-55B"][1], rows["RT-2-PaLM-E-12B"][1]
for name, a, b in zip(names, pali, palm):
    print(f"  {name:18s} {b - a:+4d}")
print("the text says PaLM-E is better on the hard scenarios and worse on the easy ones")

print()
print("RT-1: unseen objects easy", rows["RT-1"][1][0], "hard", rows["RT-1"][1][1], "(hard is higher than easy)")

Output:

model              | mean of six unseen | printed
R3M                |              11.67 |      12 | ok
VC-1               |              10.00 |      10 | ok
RT-1               |              32.33 |      32 | ok
MOO                |              34.50 |      35 | ok
RT-2-PaLI-X-55B    |              62.33 |      62 | ok
RT-2-PaLM-E-12B    |              62.50 |      62 | ok

ranking of the four baselines by unseen average: [(35, 'MOO'), (32, 'RT-1'), (12, 'R3M'), (10, 'VC-1')]
RT-2-PaLI-X-55B / RT-1 = 1.9x
RT-2-PaLI-X-55B / MOO  = 1.8x
RT-2-PaLI-X-55B / R3M  = 5.2x
RT-2-PaLI-X-55B / VC-1 = 6.2x

PaLM-E minus PaLI-X, by scenario (points):
  objects easy        +14
  objects hard        +14
  backgrounds easy    -21
  backgrounds hard    +23
  environments easy   -27
  environments hard    -2
the text says PaLM-E is better on the hard scenarios and worse on the easy ones

RT-1: unseen objects easy 31 hard 43 (hard is higher than easy)

Arithmetic on the printed table.

Evidence in the paper
§4.1 · Figure 4 · Table 4 (Appendix H.1)
Check your understanding: What does it tell you that RT-2 and RT-1 are level on seen tasks?

That web pretraining does not make the robot better at what it was trained on; the benefit appears when objects, backgrounds, environments or instructions change.

Emergent abilities: symbols, reasoning and people (Table 5)

How it works
Three groups of instructions test abilities that the robot data does not contain. Symbol understanding asks the robot to move an object to a number, an icon or a letter, such as move apple to 3. Reasoning needs sums, logos, nutrition, colours or another language, for instance pick a healthy drink. Person recognition asks it to move an object to a celebrity or to a person described by a feature, such as move coke can to person with glasses. The best model, RT-2-PaLI-X-55B, averages 60%, more than three times RT-1’s 17%. The 12-billion-parameter PaLM-E model is better on math and slightly on logos, and the authors attribute this to its different pretraining mixture.
Technical details
Table 5 (group averages), success rate in percent, as printed. The full table has nine sub-groups.
ModelSymbolsReasoningPeopleAverage
VC-111101311
RT-116162017
RT-2-PaLI-X-55B82465360
RT-2-PaLM-E-12B36434340

Our calculation Since each instruction was run five times, each group’s trial count is five times its number of instructions, and all printed percentages are whole numbers of successes. With 190 trials per model the uncertainty is wide for small groups: PaLM-E’s lead on math is 7 successes against 5 out of 20, nothing a statistical test could separate from chance.

Run the example: Table 5 as trial counts, with uncertainty
Our calculation

What each part does

  1. Compute each group’s number of trials from its number of instructions.
  2. Convert printed percentages to numbers of successes and check they are whole numbers.
  3. Give 95% intervals for the pooled success rates and test the math difference.
import math

# Table 5 (emergent evaluations). Each instruction was run 5 times, so a group's number of trials is 5 x its number of instructions.
instructions = {"Symbol 1": 3, "Symbol 2": 4, "Symbol 3": 3, "Math": 4, "Logos": 5, "Nutrition": 5, "Color/Multilingual": 8, "Celebrities": 3, "CelebA": 3}
trials = {g: 5 * n for g, n in instructions.items()}
printed = {   # success in percent, as printed
    "RT-1":            {"Symbol 1": 27, "Symbol 2": 20, "Symbol 3": 0,  "Math": 5,  "Logos": 0,  "Nutrition": 32, "Color/Multilingual": 28, "Celebrities": 20, "CelebA": 20},
    "RT-2-PaLI-X-55B": {"Symbol 1": 93, "Symbol 2": 60, "Symbol 3": 93, "Math": 25, "Logos": 52, "Nutrition": 48, "Color/Multilingual": 58, "Celebrities": 53, "CelebA": 53},
    "RT-2-PaLM-E-12B": {"Symbol 1": 67, "Symbol 2": 20, "Symbol 3": 20, "Math": 35, "Logos": 56, "Nutrition": 44, "Color/Multilingual": 35, "Celebrities": 33, "CelebA": 53},
}
total = sum(trials.values())
print("trials per model:", total, "| groups:", trials)

def wilson(k, n, z=1.96):
    # 95% Wilson score interval for a success rate k out of n
    p = k / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half

print()
print("do the printed percentages match whole numbers of successes out of 5 x instructions?")
counts = {}
for model, row in printed.items():
    counts[model] = {g: round(row[g] * trials[g] / 100) for g in row}
    ok = all(abs(100 * counts[model][g] / trials[g] - row[g]) < 0.6 for g in row)
    k = sum(counts[model].values())
    low, high = wilson(k, total)
    print(f"  {model:16s} all consistent: {ok} | pooled {k}/{total} = {100 * k / total:4.1f}% (95% interval {100 * low:4.1f} to {100 * high:4.1f})")

def fisher_two_sided(a, n1, b, n2):
    # exact two-sided Fisher test for two success counts
    m, N = a + b, n1 + n2
    prob = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p_obs = prob(a)
    return sum(prob(x) for x in range(max(0, m - n2), min(m, n1) + 1) if prob(x) <= p_obs + 1e-12)

print()
a, b = counts["RT-2-PaLM-E-12B"]["Math"], counts["RT-2-PaLI-X-55B"]["Math"]
print(f"math: PaLM-E {a}/20 against PaLI-X {b}/20, exact two-sided p = {fisher_two_sided(a, 20, b, 20):.2f}")
a, b = counts["RT-2-PaLI-X-55B"]["Symbol 2"], counts["RT-1"]["Symbol 2"]
print(f"symbol 2: PaLI-X {a}/20 against RT-1 {b}/20, exact two-sided p = {fisher_two_sided(a, 20, b, 20):.3f}")

Output:

trials per model: 190 | groups: {'Symbol 1': 15, 'Symbol 2': 20, 'Symbol 3': 15, 'Math': 20, 'Logos': 25, 'Nutrition': 25, 'Color/Multilingual': 40, 'Celebrities': 15, 'CelebA': 15}

do the printed percentages match whole numbers of successes out of 5 x instructions?
  RT-1             all consistent: True | pooled 34/190 = 17.9% (95% interval 13.1 to 24.0)
  RT-2-PaLI-X-55B  all consistent: True | pooled 109/190 = 57.4% (95% interval 50.3 to 64.2)
  RT-2-PaLM-E-12B  all consistent: True | pooled 76/190 = 40.0% (95% interval 33.3 to 47.1)

math: PaLM-E 7/20 against PaLI-X 5/20, exact two-sided p = 0.73
symbol 2: PaLI-X 12/20 against RT-1 4/20, exact two-sided p = 0.022

Our calculation; the trial counts assume five runs per instruction, as the paper states.

Evidence in the paper
§4.2 · Figure 6a · Table 5 (Appendix H.2) · Table 3 (Appendix F.2)
Check your understanding: Why is “more than three times RT-1” a stronger claim than “PaLM-E has an edge on math”?

The first compares 60% with 17% over 190 trials each, a gap far larger than the sampling noise. The second compares 7 successes with 5 out of 20, which chance can easily produce.

What matters: size and how the model is trained (Table 6)

How it works
The authors compare three training routines for RT-2-PaLI-X at two sizes: training from scratch without any web pretraining, fine-tuning a pretrained model on robot data only, and co-fine-tuning. Training the 5-billion-parameter model from scratch gives very poor results (average 9%), so they skip the 55-billion model from scratch. Co-fine-tuning generalises better than fine-tuning, and the bigger model better than the smaller one.
Technical details
Table 6, success rate in percent, as printed. The seen-task evaluation is left out of this comparison.
ModelTrainingObjects easyObjects hardBackgrounds easyBackgrounds hardEnvironments easyEnvironments hardAverage
5Bfrom scratch010460009
5Bfine-tuning24387950362342
5Bco-fine-tuning60386729442444
55Bfine-tuning60627538571952
55Bco-fine-tuning70629648633563

Our calculation The example below compares co-fine-tuning with fine-tuning scenario by scenario. At 55 billion parameters co-fine-tuning is better in five of six scenarios and equal in one, 11 points on average. At 5 billion it is better in three, equal in one and worse in two, and the averages differ by 2 points.

Run the example: re-check Table 6
Our calculation

What each part does

  1. Type in Table 6.
  2. Recompute each average and compare it with the printed one.
  3. Subtract fine-tuning from co-fine-tuning in every scenario, at both sizes.
# Table 6 (size and training strategy), typed in as printed. Six unseen scenarios, as in Table 4, then the printed average.
rows = {
    "5B  from scratch":     ([ 0, 10, 46,  0,  0,  0],  9),
    "5B  fine-tuning":      ([24, 38, 79, 50, 36, 23], 42),
    "5B  co-fine-tuning":   ([60, 38, 67, 29, 44, 24], 44),
    "55B fine-tuning":      ([60, 62, 75, 38, 57, 19], 52),
    "55B co-fine-tuning":   ([70, 62, 96, 48, 63, 35], 63),
}
names = ["objects easy", "objects hard", "backgrounds easy", "backgrounds hard", "environments easy", "environments hard"]
print("row                  | mean of six | printed")
for name, (six, printed) in rows.items():
    print(f"{name:20s} | {sum(six) / 6:11.2f} | {printed:7d}")
print("the same 55B co-fine-tuned model has the average 62 in Table 4 and 63 in Table 6")

for size in ("5B ", "55B"):
    ft, co = rows[size + " fine-tuning"][0], rows[size + " co-fine-tuning"][0]
    print()
    print(f"{size}: co-fine-tuning minus fine-tuning, by scenario (points)")
    for name, a, b in zip(names, co, ft):
        print(f"  {name:18s} {a - b:+4d}")
    better = sum(a > b for a, b in zip(co, ft))
    same = sum(a == b for a, b in zip(co, ft))
    print(f"  co-fine-tuning better in {better} of 6, equal in {same}; difference of the averages {rows[size + ' co-fine-tuning'][1] - rows[size + ' fine-tuning'][1]} points")

Output:

row                  | mean of six | printed
5B  from scratch     |        9.33 |       9
5B  fine-tuning      |       41.67 |      42
5B  co-fine-tuning   |       43.67 |      44
55B fine-tuning      |       51.83 |      52
55B co-fine-tuning   |       62.33 |      63
the same 55B co-fine-tuned model has the average 62 in Table 4 and 63 in Table 6

5B : co-fine-tuning minus fine-tuning, by scenario (points)
  objects easy        +36
  objects hard         +0
  backgrounds easy    -12
  backgrounds hard    -21
  environments easy    +8
  environments hard    +1
  co-fine-tuning better in 3 of 6, equal in 1; difference of the averages 2 points

55B: co-fine-tuning minus fine-tuning, by scenario (points)
  objects easy        +10
  objects hard         +0
  backgrounds easy    +21
  backgrounds hard    +10
  environments easy    +6
  environments hard   +16
  co-fine-tuning better in 5 of 6, equal in 1; difference of the averages 11 points

Arithmetic on the printed table.

Evidence in the paper
§4.3 · Figure 6b · Table 6 (Appendix H.3)
Check your understanding: What does the from-scratch result with 46% on easy backgrounds suggest about that scenario?

That the easy-background tasks mostly use objects the robot data already contains, so a model with no web knowledge can still do some of them; the unseen objects (0% and 10%) are where web pretraining matters.

Chain of thought in the examples

How it works
The chain-of-thought variant is shown in rollouts: for I need to hammer a nail it answers with a rock; for an instruction to bring a drink it writes a plan that names a specific can, and for an instruction to move the green objects together it plans to move the green can near the green bag of chips. Each rollout prints a plan followed by eight action numbers.
Technical details
The paper reports no success rate for this variant and no comparison with a model that does not write a plan. It describes the result as initial evidence that language-model planning can be combined with low-level control in a single model.
Evidence in the paper
§4.4 · Figures 7 and 10 · Appendix I
Check your understanding: What would turn these examples into evidence?

A fixed set of reasoning instructions run many times, with and without the plan step, and the success rates reported side by side.

An open benchmark: Language-Table (Table 1)

How it works
To compare with open-source baselines, the authors train a smaller 3-billion-parameter PaLI model on the simulated Language-Table pushing benchmark. The action is written as two numbers, each between −10 and +10, for the 2D movement of the end effector. The model is co-fine-tuned on five related tasks, including predicting the instruction and the arm position, and it runs at the same rate (5 Hz) as the baselines. It reaches 90 ± 10 against 72 ± 3 for BC-Zero, 74 ± 13 for RT-1 and 77 ± 4 for LAVA. Qualitative rollouts show pushing tasks with new objects, and failures when objects behave differently from the training blocks (a pen rolls off the table; a banana’s centre of mass is far from the contact).
Technical details

Our calculation The paper does not define the ± values or say how many runs they come from. If they were standard errors, the 13-point lead over LAVA would be 1.2 combined spreads, weak evidence, and the printed ranges (80 to 100 and 73 to 81) overlap.

Run the example: the printed spreads
Our calculation

What each part does

  1. Type in Table 1.
  2. Compare the best baseline with RT-2 using the printed spreads.
  3. Express the lead in combined spreads, assuming they are standard errors.
import math

# Table 1 (Language-Table simulation), success rate in percent as printed: mean +- the printed spread.
# The paper does not say what the spread is (standard deviation, standard error, number of runs).
rows = {"BC-Zero": (72, 3), "RT-1": (74, 13), "LAVA": (77, 4), "RT-2-PaLI-3B": (90, 10)}
best_baseline = max((m for n, (m, s) in rows.items() if n != "RT-2-PaLI-3B"))
name_best = [n for n, (m, s) in rows.items() if m == best_baseline][0]
rt2_m, rt2_s = rows["RT-2-PaLI-3B"]
lava_m, lava_s = rows[name_best]

print(f"RT-2-PaLI-3B {rt2_m} +- {rt2_s} against the best baseline {name_best} {lava_m} +- {lava_s}")
print(f"difference of the means: {rt2_m - lava_m} points")
print(f"intervals as printed: {rt2_m - rt2_s} to {rt2_m + rt2_s} and {lava_m - lava_s} to {lava_m + lava_s}  (they overlap)")

# If each spread were a standard error, how many combined spreads is the difference?
combined = math.sqrt(rt2_s ** 2 + lava_s ** 2)
print(f"if the spreads were standard errors: difference / combined spread = {(rt2_m - lava_m) / combined:.1f}")
print("if they were standard deviations over n runs the gap would be larger by about the square root of n; n is not given")
print("RT-1 and RT-2 differ by", rt2_m - rows["RT-1"][0], "points with spreads", rows["RT-1"][1], "and", rt2_s)

Output:

RT-2-PaLI-3B 90 +- 10 against the best baseline LAVA 77 +- 4
difference of the means: 13 points
intervals as printed: 80 to 100 and 73 to 81  (they overlap)
if the spreads were standard errors: difference / combined spread = 1.2
if they were standard deviations over n runs the gap would be larger by about the square root of n; n is not given
RT-1 and RT-2 differ by 16 points with spreads 13 and 10

Arithmetic on the printed table; what the spread means is not stated.

Evidence in the paper
§4.1 “Open Source Language Table Benchmark” · Table 1 · Figures 5 and 9 · Appendix B, D and G
Check your understanding: Why does it matter that the paper does not say what “±” means?

Without knowing whether it is a standard deviation, a standard error or a range, and over how many runs, the size of the lead cannot be judged.

What the results show

  • With the same robot data, a pretrained vision-language model fine-tuned to write actions does about twice as well as RT-1 and MOO on new objects, backgrounds and environments.
  • It follows instructions with symbols, simple reasoning and people that the robot data does not contain, with a clear margin over RT-1 and VC-1.
  • In the 55-billion-parameter ablation, co-fine-tuning beats fine-tuning on robot data alone, and both beat training from scratch.
  • The same idea works in a simulated benchmark with a smaller model.

What they do not show

  • Any gain on tasks seen in training, or any new motion: the robot only reuses the skills in its data.
  • That the margins are free of noise: no repeated runs or intervals are given, and many groups have 15 to 40 trials.
  • That co-fine-tuning helps at every size: at 5 billion parameters the gain is small and mixed.
  • That chain-of-thought helps: it is shown only through examples.
  • That the results hold for other robots, tasks that need fast reactions or dexterous manipulation.

Part E · optional

Engineering deep dive

The action string field by field, the training objective, the baselines, the evaluation instruction groups and the failure cases, for readers who want to rebuild or audit the work. Everything is collapsed; Parts A to D do not depend on it.

The action string, field by field
Section 3.2. Written in this order with spaces between the numbers. In every eight-number string the paper prints, the first number is 1; the last is 127 in five of the six and 255 in the other.
PositionMeaningValues
1terminate: ends the episode when the policy judges the task donediscrete command, not binned; the paper does not say which value means stop
2 to 4change in position of the end effector, x, y, zone of 256 bins each
5 to 7change in rotation of the end effector, x, y, zone of 256 bins each
8extension of the gripperone of 256 bins

Our calculation The last sentence is our observation from the paper’s printed examples (see F1).

The training objective (our formulation)

Our interpretation The paper states in words that training uses next-token prediction, equal to the behaviour-cloning loss. Written out, for one example with answer tokens y1 to yn (the eight numbers, or a web answer):

L(θ)=−∑i=1…n log⁡ pθ(yi ∣image, instruction, y1, …, yi−1)L(\theta )=-\sum _{i=1\ldots n}\,\log \,p_{\theta }(y_{i}\,\mid \text{image},\,\text{instruction},\,y_{1},\,\ldots ,\,y_{i-1})

averaged over a mixture in which robot examples make up about 50% (PaLI-X) or 66% (PaLM-E) of each batch. In the chain-of-thought variant the answer tokens are the plan followed by the action.

The baselines, in detail
  • Reported by the paper RT-1: 35-million-parameter transformer trained on the robot data only; the check on whether web pretraining matters.
  • Reported by the paper VC-1: a ViT-L visual representation made for robotics. It has no language input, so the instruction is embedded separately with a universal sentence encoder, concatenated with the image tokens, passed through TokenLearner and fed to an RT-1 decoder; the weights are unfrozen because frozen ones were far worse.
  • Reported by the paper R3M: as VC-1, with a ResNet50 learned from the Ego4D videos of human activity, also unfrozen.
  • Reported by the paper MOO: a vision-language model marks the object of interest with a single coloured pixel, and an end-to-end policy uses the marked image; the model is used as a perception aid, not for the policy’s own representation.
Evaluation instruction groups for the emergent tests (Table 3)
Our summary of Table 3 and Appendix F.1; each instruction was run 5 times.
GroupInstructionsKind of instruction
Symbol 1, 2, 33, 4, 3move an object near or onto letters, numbers, drawings or icons
Reasoning: math4sums, products and “the smallest number”
Reasoning: logos5move a cup to a company logo or to “a search engine”
Reasoning: nutrition5healthy or sweet or salty items
Reasoning: colour and multilingual8matching colours, and instructions in Spanish, German and French
People: celebrities, CelebA3, 3move an object to a named celebrity or to a person with a described feature
Failure cases (Appendix G)

Reported by the paper In Language-Table, the model attends to the right object but cannot control the unfamiliar object dynamics. On the real robot the authors name four weak spots: grasping an object by a specific part such as a handle, motions never seen in the robot data (wiping with a towel, tool use), dexterous or precise motions such as folding a towel, and reasoning with several layers of indirection.

Design choices that the paper does not test
  • The size of the robot-data share (about 50% and 66%) is stated but not varied.
  • The 256-bin discretisation is inherited from RT-1; fewer or more bins are not compared.
  • Writing one action per query is not compared with writing several steps at once.

Our interpretation Our reading of what the paper contains; later guides (π0, AsyncVLA) revisit some of these choices.

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: where its numbers disagree with each other, where the wording goes further than the evidence and what the experiments cannot show. RT-2 is an influential paper and its main result, that web pretraining helps generalisation, is supported by the tables; the findings below mostly concern details and strength of evidence. Each one links back to the concept it affects, and the calculations are reproducible in the examples.

Numbers that do not agree

  1. F1. The printed action string has seven numbers, not eight Numerical inconsistency

    What the paper says. An action is eight whole numbers: the stop command followed by seven values. The example target in Section 3.2 is a string of numbers.

    What we found. The example string in Section 3.2 contains seven numbers, and the one in Figure 1 contains six (it is a schematic). The six real outputs printed in Section 4.4 and Figure 7 all have eight. The authors’ blog post prints the Section 3.2 string with an eighth number at the end.[2] In all six eight-number strings the first number is 1, and the last is 127 in five of them and 255 in the sixth. The paper never says which value of the first number means “stop”.

    Why it matters. A reader who tries to rebuild the format from the printed example gets a string of the wrong length, and cannot tell from the paper which value ends the episode. Concept: Actions as text · example

    Run the example: counting the numbers in the printed action strings
    Our calculation

    What each part does

    1. Type in every action string that the paper prints.
    2. Count the numbers in each and look at the first and last one.
    3. List the strings that do not have eight numbers.
    # Count the numbers in every action string that the paper prints, and look at the first and last number.
# The paper says an action is 8 integers: one termination command followed by seven continuous values
# (three position changes, three rotation changes, the gripper extension).
printed = {
    "Section 3.2 example":          "1 128 91 241 5 101 127",
    "Figure 1 example":             "132 114 128 5 25 156",
    "Section 4.4 chain of thought": "1 128 124 136 121 158 111 255",
    "Figure 7, hammer":             "1 129 138 122 132 135 106 127",
    "Figure 7, green objects":      "1 130 129 121 131 127 128 127",
    "Figure 7, different object":   "1 128 129 125 131 125 128 127",
    "Figure 7, all objects":        "1 128 126 127 135 123 119 127",
    "Figure 7, bring me a drink":   "1 143 129 123 145 114 115 127",
}
print(f"{'string':30s} | numbers | first | last")
for name, text in printed.items():
    numbers = [int(x) for x in text.split()]
    print(f"{name:30s} | {len(numbers):7d} | {numbers[0]:5d} | {numbers[-1]:4d}")

expected = 8
wrong = [name for name, text in printed.items() if len(text.split()) != expected]
print()
print("strings that do not have", expected, "numbers:", wrong)

    Output:

    string                         | numbers | first | last
Section 3.2 example            |       7 |     1 |  127
Figure 1 example               |       6 |   132 |  156
Section 4.4 chain of thought   |       8 |     1 |  255
Figure 7, hammer               |       8 |     1 |  127
Figure 7, green objects        |       8 |     1 |  127
Figure 7, different object     |       8 |     1 |  127
Figure 7, all objects          |       8 |     1 |  127
Figure 7, bring me a drink     |       8 |     1 |  127

strings that do not have 8 numbers: ['Section 3.2 example', 'Figure 1 example']

    Counting on strings printed in the paper; the strings are typed in by us.

  2. F2. The same model has average 62 in Table 4 and 63 in Table 6 Numerical inconsistency

    What the paper says. Table 4 gives RT-2-PaLI-X-55B an unseen average of 62. Table 6 gives the 55B co-fine-tuned model an average of 63.

    What we found. The six scenario values are identical in the two tables (70, 62, 96, 48, 63, 35), and their mean is 62.33. The two averages cannot both be roundings of the same number.

    Why it matters. A small slip, probably in Table 6, but it affects the headline gap between co-fine-tuning and fine-tuning at 55B (63 − 52 = 11 points, or 10 with the other value). Concept: Size and training ablation · example

  3. F3. “Over 280 tasks” against 138 listed instructions Numerical inconsistency

    What the paper says. The unseen evaluations consist of over 280 tasks; the instructions are listed in Table 2, and each was run between 1 and 5 times.

    What we found. Counting Table 2 gives 138 instructions: 10 and 21 for objects, 12 and 17 for backgrounds, 45 and 33 for environments. At about two runs per task, the whole evaluation adds up to the reported 6,000 trials (1.99 on average), so “tasks” may count repeated runs, but the paper does not say. The number of runs per instruction is not given, so the number of trials behind each cell of Table 4 is unknown.

    Why it matters. Without the trial counts one cannot attach uncertainty to Tables 4 and 6. For Table 5 it can be derived, because every instruction was run five times. Concept: The set-up · example

  4. F4. “Seven skills” and a list of eight Numerical inconsistency

    What the paper says. Appendix B says each demonstration is annotated with an instruction from one of seven skills.

    What we found. The list that follows has eight entries: pick, move near, place upright, knock over, open drawer, close drawer, place into a receptacle, and pick from a receptacle and place on the counter. Section 4.1 groups the seen tasks into six families, where opening and closing drawers are one.

    Why it matters. A very small inconsistency of counting, but it shows that the description of the robot data in this paper is loose and defers to the RT-1 paper for details. Concept: The data

  5. F5. A model called 3 billion parameters whose parts add up to 5 billion Numerical inconsistency

    What the paper says. The model for Language-Table is called PaLI-3B and is described as a vision transformer of 2 billion parameters and a language network called UL2-3B.

    What we found. The two listed parts add up to 5 billion, not 3 billion. For PaLI-X the same arithmetic works: 22 billion plus 32 billion gives 54 billion, close to the name 55B. Possibly the “3B” excludes the vision part, but the paper does not say.

    Why it matters. Parameter counts are the paper’s main way of describing model scale, so it is worth knowing whether “3B” counts everything. Concept: The backbones · example

How far the claims go

  1. F6. The emergent tests omit the best baseline, and say why differently from Table 4 Ambiguity

    What the paper says. For the emergent evaluations the authors say they use the top two baselines of the earlier evaluations, RT-1 and VC-1, and compare them with RT-2.

    What we found. In Table 4 the baselines rank MOO (35), RT-1 (32), R3M (12) and VC-1 (10) by unseen average, and the text of Section 4.1 itself names RT-1 and MOO as the next two. VC-1 is the lowest of the four. MOO, the baseline that also uses a vision-language model, is not in Table 5.

    Why it matters. RT-1 is a sensible baseline, but the comparison most relevant to “does a vision-language model help?” is with MOO, and the paper does not say why it was left out (it may not accept the instructions, which the paper does not state). Concept: Emergent abilities · example

  2. F7. “Better on the harder scenarios, worse on the easier ones” Ambiguity

    What the paper says. PaLM-E seems to perform better in harder versions of the generalisation scenarios while under-performing on easier ones, giving a similar average.

    What we found. Against PaLI-X-55B, PaLM-E-12B is ahead by 14 points on easy objects and 14 on hard objects, 23 on hard backgrounds, behind by 21 on easy backgrounds and 27 on easy environments, and 2 behind on hard environments. So the pattern holds on backgrounds and environments but not on objects, and hard environments are a tie. A footnote adds that PaLM-E’s original pretraining includes robot images for planning questions that can resemble the generalisation scenes, though without low-level actions.

    Why it matters. The sentence describes three of six scenarios; the averages are equal because the large gaps in both directions cancel, and the footnote is a possible advantage for the unseen environments. Concept: Generalisation · example

  3. F8. “Regardless of size” co-fine-tuning is better Method caveat

    What the paper says. Co-fine-tuning, regardless of the model’s size, results in better generalisation than fine-tuning with robot data only.

    What we found. At 55 billion parameters this holds in five of six scenarios (one tie), by 11 points on average. At 5 billion parameters it holds in three, ties in one and loses on easy backgrounds (by 12 points) and hard backgrounds (by 21 points); the averages differ by 2 points, 44 against 42, with no repeated runs.

    Why it matters. The evidence for co-fine-tuning is clear at 55 billion parameters and weak at 5 billion; “regardless of size” rests on one size. Concept: Size and training ablation · example

  4. F9. Size is compared with different amounts of training Ambiguity

    What the paper says. The increased size of the model results in better generalisation.

    What we found. The 5B model was co-fine-tuned for 270,000 steps and the 55B model for 80,000 steps, both at batch size 2,048, so the larger model saw about 3.4 times fewer training examples and still scored higher (63 against 44). The paper does not comment on the different budgets, which were presumably chosen per model.

    Why it matters. The conclusion is plausible, and the difference works against the larger model, but the comparison is not at equal training, so it cannot separate the effect of size from the effect of the training schedule. Concept: Training settings · example

  5. F10. “Keeping the original data around” is a small slice, and forgetting is not measured Method caveat

    What the paper says. Co-fine-tuning keeps the original data around so that the model does not forget the concepts it learned from vision-language training.

    What we found. With the stated steps, batch size and mixture, the 55B model sees about 81.9 million web examples, under a tenth of the 1 billion filtered WebLI examples, so the web data continues as a stream, not a replay. The paper does not report how well RT-2 still answers web questions after co-fine-tuning, either, so forgetting is inferred from the robot results and not measured.

    Why it matters. The recipe works as reported on robot generalisation, but the explanation (forgetting) is a hypothesis. Whether the model also stays a good vision-language model is not shown. Concept: Co-fine-tuning · example

  6. F11. No repeated runs, and differences of two or three trials Method caveat

    What the paper says. RT-2 outperforms the baselines by a large margin; PaLM-E has an edge on math; the larger PaLI-X model is better on symbols, reasoning and person recognition on average.

    What we found. For the emergent tests we can count trials: 190 per model. The pooled success of RT-2-PaLI-X-55B is 109 of 190 (57%, 95% interval about 50 to 64%) against 34 of 190 (18%, 13 to 24%) for RT-1, a clear gap. But the math group has 20 trials: 7 successes for PaLM-E against 5 for PaLI-X, with an exact test p-value of 0.73, and reasoning averages of 46 against 43 are of the same size. The printed “average” is the plain mean of group averages (60), not the pooled rate (57.4%).

    Why it matters. The headline gaps are real; the finer statements about which model is better at what are within the noise that the trial counts allow. Concept: Emergent abilities · example

  7. F12. Language-Table: an undefined spread and an unequal comparison Ambiguity

    What the paper says. There is a significant performance boost compared with the baselines on Language-Table: 90 ± 10 against 72 ± 3, 74 ± 13 and 77 ± 4.

    What we found. The paper does not say what the spread is or how many runs it comes from. The best baseline’s range (73 to 81) touches RT-2’s (80 to 100); if the spreads were standard errors the lead would be 1.2 combined spreads. Appendix B also says that the RT-2 model was trained on five tasks, four of them auxiliary (predicting the instruction, the arm position, the time between frames and success); the paper does not say whether the baselines had such extra supervision. The caption of Table 1 cites a different Lynch paper (2020) from the text (2022).

    Why it matters. The result is encouraging, but “significant” is not backed by a test, and the lead may partly come from extra training signals the baselines did not use. Concept: Language-Table · example

  8. F13. Chain of thought is shown, not measured Limitation

    What the paper says. Adding chain-of-thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example choosing a rock to hammer with and an energy drink for someone who is tired.[1]

    What we found. The evidence is a handful of rollouts (Figures 7 and 10) from a variant fine-tuned for just a few hundred gradient steps. There is no success rate, number of trials or comparison with the same model without a plan, and the hammer example in Figure 7 prints the answer “rocks” without a plan.

    Why it matters. The result is an existence proof of the idea, in the paper’s own words “initial evidence”; it does not show how often the plan helps, or whether it ever misleads the action. Concept: Chain of thought · in the examples

The paper’s own limits

  1. F14. What the paper says about its limits, and what it leaves out Limitation

    What the paper says. Section 5 lists three limits: the robot gains no new motions from web data; large models are expensive to run, so fast control may be a bottleneck; and few vision-language models are generally available for building VLAs. Appendix G lists weak spots.[1]

    What we found. These are accurate and are borne out by Part D: no gain on seen tasks, and a rate of 1 to 3 Hz for the largest model. The paper does not discuss that the evidence comes from one lab and one robot platform with its own task suite (over 280 tasks “focus primarily on pick and placing”), that the models are trained on proprietary data and weights that outside readers cannot rerun, or what the success rates on the hardest scenarios (about a third for unseen environments) would mean in use. The abstract’s emphasis on reasoning and emergent skills sits beside absolute emergent success rates of 60% for the best model and 40% for the other.

    Why it matters. RT-2 shows that the idea works and gives a first estimate of how much it helps; it is not evidence that the recipe gives a reliable general-purpose controller. Concept: Running a very large model · Generalisation

Knowledge check

Try each question first, then reveal the explanation.

Question 1

In your own words: how does RT-2 make a vision-language model control a robot?

Explanation

It writes each action as eight whole numbers in text: a stop command, three movements, three rotations and the gripper, each in one of 256 slots. The model is fine-tuned on robot demonstrations and web questions together, and at run time the numbers it writes are converted back into a movement.

Question 2

Why does RT-2 train on web data and robot data together?

Explanation

Training on robot data alone lets the model drift towards the narrow robot data. Keeping web question answering in every batch preserved generalisation to new objects and instructions in the paper’s ablation, which the authors attribute to the model not forgetting its web concepts.

Question 3

What does RT-2 not do, according to the paper?

Explanation

It does not learn new motions: its physical skills stay within the robot data, though it can use them in new ways. It is also slow (1 to 3 Hz for the 55-billion-parameter model) and needs a cloud service.

Question 4

Name one inconsistency in the paper and one claim stronger than the evidence.

Explanation

For example: Table 4 and Table 6 give 62 and 63 as the average of the same model; or the Section 3.2 example action string has seven numbers where eight are described. And: the chain-of-thought result is shown only through example rollouts, with no success rate or comparison.

Glossary

Vision-language model (VLM)
A model that reads images and text and writes text, for example answering a question about a picture.
Vision-language-action model (VLA)
A vision-language model that also writes robot actions, here as text tokens.
Visual question answering (VQA)
The task of answering a written question about an image.
Token
A small piece of text, such as a word part or a number, that a language model reads and writes.
Action token
A token that stands for one of the 256 slots of a movement value.
Discretisation / bin
Cutting a continuous range into equal slices; each slice is a bin with a number.
End effector
The part of the robot that acts on things, here the hand and gripper.
6-DoF
Six degrees of freedom: three directions of movement and three of rotation.
Closed-loop control
Choosing each action after looking at the latest camera image.
Fine-tuning
Continuing to train a pretrained model on a new, smaller data set.
Co-fine-tuning
Fine-tuning on the new data and the original data mixed together.
Symbol tuning
Training a model to use existing tokens as symbols with a new meaning.
Output constraint
Allowing only certain tokens when the model writes its answer.
Behaviour cloning
Learning by imitating the actions in demonstrations.
Next-token prediction
The training task of predicting each token of the answer from the ones before it.
Chain of thought
Writing out reasoning steps in words before the final answer.
Emergent capability
An ability that appears from pretraining rather than from the robot demonstrations.
Generalisation
Doing well on objects, backgrounds, environments or instructions not seen in training.
Baseline
A method used for comparison, trained on the same robot data.
Hz
Times per second: here, how many actions the robot receives each second.
TPU
Tensor processing unit: Google’s chip for training and running large models.

References

Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

  1. Brohan, A., et al. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control. Conference on Robot Learning (CoRL) 2023. arXiv:2307.15818

    Cited for:

    • Introduces the name vision-language-action (VLA) models for vision-language models co-fine-tuned on robot trajectories and Internet-scale vision-language tasks. Checked against the paper’s abstract.
    • Fine-tunes existing vision-language models on both robotic trajectory data and Internet-scale vision-language tasks such as visual question answering (co-fine-tuning), with actions expressed as text tokens in the same way as natural-language tokens. Checked against the paper’s abstract.
    • Instances were built on the PaLI-X (55B) and PaLM-E (12B) vision-language models. Checked against the paper, via summaries of its text.
    • In about 6,000 evaluation trials, the models showed better generalisation to novel objects, could follow commands absent from the robot training data (for example placing an object on a specific number or icon) and handled basic reasoning such as picking the smallest object or the one nearest another. Checked against the paper’s abstract.
    • Writes robot actions as text tokens so that the model handles them in the same way as language. Checked against the paper’s abstract.
    • The model’s physical skills remain limited to the distribution of skills seen in the robot data, although it can apply them in new ways using web knowledge. Checked against the paper’s abstract.
    • Adding chain-of-thought reasoning enabled multi-step semantic reasoning, such as choosing an improvised tool or picking a drink for a tired person. Checked against the paper’s abstract.

  2. Google DeepMind (2023). RT-2: New model translates vision and language into action. Google DeepMind blog. deepmind.google

    Cited for:

    • Shows an action written as a string of integers, for example “1 128 91 241 5 101 127 217”, in which the first number says whether to continue or terminate the episode. Checked against the authors’ own blog post.

Go further

All paper guides · Back to the VLA track