VLA · Paper guide

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

The dataset that Octo and OpenVLA train on, and the first test of whether pooling robots helps, explained through the questions an engineer would ask.

Open X-Embodiment Collaboration (Padalkar, Pooley, Jain, Bewley et al.)
arXiv:2310.08864 (v9)
Overview: about 4 minutes
Full guide: about 40 minutes

This is a guide to the Open X-Embodiment paper, not a sentence-by-sentence paraphrase. It is a dataset-and-models paper: the “method” is a data format and a set of alignment choices, and the models are RT-1 and RT-2 from earlier papers. It is the data counterpart to the model guides: Octo trains on 25 of its datasets and OpenVLA on 27, and DROID uses it as a baseline. Part A is a 4-minute overview. Parts B to D explain the data, the models and the experiments one idea at a time. Tables and formats are folded away in Part E, and Part F is a critical review of the paper’s numbers and claims.

Start with the overview Jump to the critical review

How to use this page

In a hurry: read Part A (about 4 minutes).
To understand the paper: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example, then technical details and where to find them in the paper.
To check or use it: open the folded sections (the data format, action tokens, the mixture, runnable examples) and read Part F.

Open the paper (arXiv:2310.08864) beside this page; section, table and figure references are to it. We paraphrase and never reproduce its text. Statements are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own small model of an idea, and are never presented as results of the paper.

Part A · about 4 minutes

Start here: the paper at a glance

The problem

Robot learning has usually meant training one model for every application, every robot and often every room. Vision and language models changed that: one big model trained on broad data became the starting point for many tasks. Robotics lacks the broad data, because each lab records its own robot, in its own format, on its own tasks. The authors ask whether the same consolidation can happen: can a single “X-robot” policy trained on data from many robots do better on each robot than a model trained on that robot’s data alone?

The main contribution

The Open X-Embodiment (OXE) dataset. 60 existing robot datasets from 34 labs (21 institutions), converted to one data format (RLDS, the reinforcement learning datasets format), with more than a million trajectories from 22 robot types and, by the authors’ count, 527 skills.^[1]
RT-X models. Two existing models, RT-1 (a small robot transformer) and RT-2 (a vision-language-action model that writes actions as text), trained on data from 9 of the robots with no new mechanism for the differences between robots: RT-1-X with 35 million parameters and RT-2-X with 55 billion.^[1]
Evidence of positive transfer. About 3,600 real-robot trials on 6 robots: RT-1-X beats each dataset’s original method on 4 of 5 small datasets, and RT-2-X does about three times better on skills that its own robot’s data never contained but other robots’ data did.^[1]

How the data and the models work, in one paragraph

Every dataset keeps its own robot, cameras and action format, so the authors align them only coarsely. One camera view per dataset is chosen and resized, and the action is converted to seven numbers for the hand (position and rotation changes or velocities, and the gripper) plus one for ending the episode. Each dataset’s actions are normalised on their own and then cut into 256 bins, so a token means something different on each robot. RT-1 takes a history of images and a sentence through a convolutional network and a transformer. RT-2 starts from a vision-language model, writes the action tokens as ordinary text, and is co-trained with web data. Both are trained with a plain cross-entropy loss and run at 3 to 10 Hz.

What it learns from, and what it is tested on

Learns from: a mixture of 12 named datasets from 9 robot types (everything available when the experiments were run); RT-2-X is also trained on web vision-language data in about equal parts. Tested on: six robots in five small-data settings (kitchen, cable routing, door opening, a UR5 arm, robot play), two large-data settings (the Bridge WidowX robot at two sites and the Google robot), and unseen-object and “emergent skill” tests on the Google robot, always against the dataset’s original method and an RT-1 trained on that dataset alone.

What the paper reports

Figure 4 and Tables I and II, as printed.
Test	RT-X	Comparison
Small datasets, mean of 5 (success, %)	63 (RT-1-X)	41 original methods; 44 RT-1 on the dataset alone
Large datasets (Bridge, Google robot), RT-1-X	27, 27, 73	RT-1 40, 30, 92; originals 13, 13, none
Large datasets, RT-2-X (55B)	50, 30, 91	RT-1 40, 30, 92
Emergent skills on the Google robot (%)	75.8 (RT-2-X)	27.3 for RT-2 trained on that robot’s data; 42.8 without the Bridge data
Design choices, 5B model (emergent skills, %)	44.4 with history and web pretraining	14.5 without history; 0.0 from scratch; 48.7 fine-tuned only

What to keep in mind

The paper gives 3,600 trials in total but no count for any bar, and the printed percentages in Table II cannot come from a single trial count.
The models were trained on 9 of the 22 robots, and all the robots tested were in the training data: the paper shows transfer between known robots, not to a new one.
The small model does not benefit on the large datasets, and the “parameter scaling” comparison between it and the 55-billion-parameter model changes architecture, pretraining and data together.
The authors say they did not study robots with very different sensing and actuation, or when transfer helps and when it hurts.

If you read nothing else: Open X-Embodiment is the dataset that made large-scale, multi-robot training practical and later models (Octo, OpenVLA) are trained on it; its experiments suggest that pooling robots’ data helps small datasets and that a big web-pretrained model can use it even on a robot that has lots of data, but the evidence is thinly documented and not a test of transfer to a new robot.

Check your understanding: Why does the paper say the RT-1 baseline “trained on the dataset in isolation” matters?

RT-1 and RT-1-X are the same network. If RT-1-X, trained on the pooled data, beats RT-1 trained on one dataset, the difference comes from the extra data and not from a better architecture. The comparison with the original method asks whether the architecture is good enough.

Part B

Understand the method

The “method” of this paper is mostly a data format and a set of coarse alignment choices, together with two models that are taken from earlier papers. Each idea below follows the same pattern, from intuition to where to find it in the paper.

One training stream. Orange marks the design choices the guide calls out as the paper’s main contribution. Drawn by us from Sections III and IV and Figures 2 and 3 of the paper.

1. The repository: 60 datasets in one format

Intuition

Turn many private folders into one library that downloads the same way.

The problem it solves

Robot datasets come in different formats, with different cameras, action spaces and metadata, so using several at once means writing new code for each.

How it works

The Open X-Embodiment repository contains the dataset and a selection of pre-trained RT-X checkpoints. The dataset pools 60 existing datasets from 34 research labs and converts them to the RLDS format, which stores serialised tfrecord files and accommodates different numbers of RGB cameras, depth cameras and point clouds, with loading in all major deep learning frameworks. The authors count more than a million trajectories from 22 embodiments, from single arms to two-armed robots and quadrupeds, and 21 institutions are involved. It is meant to be a community effort that keeps growing.

Example

Illustration, not from the paper A lab with a wrist camera and a depth sensor and a lab with a single third-person camera both load their data through the same function and get episodes of observations, actions and a sentence.

Why the authors chose it

The authors say the goal is to organise large datasets to enable research on X-embodiment models, and to show that robot datasets can be shared in a meaningful and useful way.

Technical details

Our calculation By our arithmetic, 1 million trajectories over 60 datasets is at least 16,667 per dataset on average, and 60 datasets over 22 embodiments is 2.7 per embodiment. The counts have moved on: a later paper’s table lists 1.4 million trajectories and another reports more than 70 datasets and more than 2 million trajectories.

Run the example: headline counts and evaluation effort

Our calculation

What each part does

Type in the counts of datasets, labs, embodiments, skills and trials.
Derive averages and compare them with the numbers in later papers.
Count how many of the datasets and embodiments are in the training mixture.

# The headline counts of the Open X-Embodiment paper, and what simple arithmetic says about them.
traj, datasets, embodiments, institutions, labs, skills, tasks = 1_000_000, 60, 22, 21, 34, 527, 160_266
print(f"'1M+' trajectories in {datasets} datasets: at least {traj / datasets:,.0f} per dataset on average (the datasets are very uneven)")
print(f"{datasets} datasets from {labs} labs and {institutions} institutions: {datasets / labs:.1f} datasets per lab, {labs / institutions:.1f} labs per institution")
print(f"{embodiments} embodiments: {datasets / embodiments:.1f} datasets per embodiment on average (Figure 2a: Franka alone has the most; many have one)")
print(f"{skills} skills and {tasks:,} tasks: {tasks / skills:.0f} tasks per skill; the paper does not define either count")
print()
mix = ["RT-1", "QT-Opt", "Bridge", "Task Agnostic Robot Play", "Jaco Play", "Cable Routing", "RoboTurk", "NYU VINN", "Austin VIOLA", "Berkeley Autolab UR5", "TOTO", "Language Table"]
print(f"the training mixture for RT-1-X and RT-2-X names {len(mix)} datasets from 9 embodiments:")
print(f"  datasets: {len(mix)} of {datasets} = {100 * len(mix) / datasets:.0f}%; embodiments: 9 of {embodiments} = {100 * 9 / embodiments:.0f}%")
print("  (the paper says the dataset kept growing, and the mixture was everything available at the time of the experiments)")
print()
print("how the dataset has grown in later papers' descriptions")
for who, t, d in (("this paper (v9)", "1M+", "60"), ("DROID's Table I (2024)", "1.4M", "n/a"), ("OpenVLA (2024)", "more than 2M", "more than 70")):
    print(f"  {who:26s} {t:14s} trajectories, {d} datasets")
print()
print("evaluation effort")
print(f"  3,600 trials over 6 robots = {3600 / 6:.0f} per robot")
cells = 15 + 11 + 14   # Figure 4 (5 domains x 3 methods), Table I (4 + 4 + 3), Table II (7 rows x 2 columns)
print(f"  printed result cells: Figure 4 has 15, Table I has 11, Table II has 14 = {cells}; if all trials were spread evenly over them: {3600 / cells:.0f} per cell")
print(f"  RT-1-X has 35 million parameters and RT-2-X 55 billion: {55e9 / 35e6:,.0f} times as many")

Output:

'1M+' trajectories in 60 datasets: at least 16,667 per dataset on average (the datasets are very uneven)
60 datasets from 34 labs and 21 institutions: 1.8 datasets per lab, 1.6 labs per institution
22 embodiments: 2.7 datasets per embodiment on average (Figure 2a: Franka alone has the most; many have one)
527 skills and 160,266 tasks: 304 tasks per skill; the paper does not define either count

the training mixture for RT-1-X and RT-2-X names 12 datasets from 9 embodiments:
  datasets: 12 of 60 = 20%; embodiments: 9 of 22 = 41%
  (the paper says the dataset kept growing, and the mixture was everything available at the time of the experiments)

how the dataset has grown in later papers' descriptions
  this paper (v9)            1M+            trajectories, 60 datasets
  DROID's Table I (2024)     1.4M           trajectories, n/a datasets
  OpenVLA (2024)             more than 2M   trajectories, more than 70 datasets

evaluation effort
  3,600 trials over 6 robots = 600 per robot
  printed result cells: Figure 4 has 15, Table I has 11, Table II has 14 = 40; if all trials were spread evenly over them: 90 per cell
  RT-1-X has 35 million parameters and RT-2-X 55 billion: 1,571 times as many

Arithmetic on the printed counts; the per-cell trial estimate is ours.

Evidence in the paper

§III · Figure 1 · Figure 2

Check your understanding: Is “22 embodiments” the same as “22 robot models”?

An embodiment here is a robot platform: a Franka arm, a Google robot, a WidowX, a Jackal and so on. Several datasets can use the same embodiment, which is why 60 datasets come from 22 embodiments.

2. Coarse alignment of inputs and actions

Intuition

Make everything the same shape, but not the same meaning.

The problem it solves

A model with a fixed input and output cannot read datasets with different cameras and action formats.

How it works

The model sees a history of recent images and a language instruction and predicts a 7-dimensional end-effector action: x, y, z, roll, pitch, yaw and gripper opening, or the rates of those. One canonical camera view is chosen from each dataset and resized to a common resolution; the original action set is converted to a 7-DoF (degree-of-freedom) end-effector action; and each dataset’s actions are normalised before discretisation, so that the output can be de-normalised differently for each embodiment. The authors do not align the coordinate frames, and action values may be absolute or relative positions or velocities, as in the original control scheme, so the same action vector may produce very different motions on different robots. Camera poses and properties also differ.

Example

Illustration, not from the paper The action tokens “1 128 91 241 5 101 127” would be a small relative move on one robot and a position on a table on another.

Why the authors chose it

It is the cheapest way to pool data: nothing needs to be re-recorded, and the model has to learn which robot it is controlling from what it sees.

Technical details

Toy illustration The example normalises three made-up datasets (relative moves, absolute positions, velocities) and decodes the same tokens for each, showing that token 128 means about zero on two of them and the middle of the table on the third. It is a toy: the paper does not publish its normalisation ranges, and does not say that the embodiment is named in the input.

Run the example: the same token, three robots

Toy illustration

What each part does

Make three datasets with different action meanings and ranges.
Cut each into 256 bins and decode the same tokens for each.
List what the model must infer to choose the right reading.

# Toy: why 'the same action vector may induce very different motions for different robots' (Section IV-A).
# Two made-up datasets, each normalised on its own, then cut into 256 bins like RT-1 and RT-2 do. Not from the paper's data.
bins = 256
datasets = {
    "robot A: relative moves (metres per step)": (-0.05, 0.05),
    "robot B: absolute positions (metres)": (0.20, 0.80),
    "robot C: velocities (metres per second)": (-0.50, 0.50),
}
def decode(token, lo, hi):
    return lo + (token + 0.5) * (hi - lo) / bins          # the bin centre, de-normalised for this dataset
tokens = [1, 128, 91, 241, 5, 101, 127]       # the example action text of the RT-2 description: "1 128 91 241 5 101 127"
print("the same seven tokens, three readings of the first number (dimension x):")
for t in (0, 64, 128, 191, 255):
    row = ", ".join(f"{name.split(':')[0]} {decode(t, *r):+.4f}" for name, r in datasets.items())
    print(f"  token {t:3d}: {row}")
print()
lo, hi = datasets["robot B: absolute positions (metres)"]
print(f"for the first token of the example, {tokens[1]} (the second entry here):")
for name, r in datasets.items():
    print(f"  {name:45s} -> {decode(tokens[1], *r):+.4f}")
print()
print("what the model has to infer: which dataset's scale to use. The paper normalises each dataset before discretising and de-normalises per embodiment;")
print("it does not say that the embodiment is named in the input, so the policy has to tell robots apart from the images.")

Output:

the same seven tokens, three readings of the first number (dimension x):
  token   0: robot A -0.0498, robot B +0.2012, robot C -0.4980
  token  64: robot A -0.0248, robot B +0.3512, robot C -0.2480
  token 128: robot A +0.0002, robot B +0.5012, robot C +0.0020
  token 191: robot A +0.0248, robot B +0.6488, robot C +0.2480
  token 255: robot A +0.0498, robot B +0.7988, robot C +0.4980

for the first token of the example, 128 (the second entry here):
  robot A: relative moves (metres per step)     -> +0.0002
  robot B: absolute positions (metres)          -> +0.5012
  robot C: velocities (metres per second)       -> +0.0020

what the model has to infer: which dataset's scale to use. The paper normalises each dataset before discretising and de-normalises per embodiment;
it does not say that the embodiment is named in the input, so the policy has to tell robots apart from the images.

Made-up numbers; it illustrates the design, not the paper’s results.

Evidence in the paper

§IV-A · Figure 3

Check your understanding: What does the model need to know that the tokens do not tell it?

Which robot, and so which normalisation, it is acting for. The paper normalises per dataset but does not describe giving the embodiment as an input, so it must be inferred from the camera image and context.

3. Actions as 256-bin tokens, eight dimensions

Intuition: The same recipe as RT-1 and RT-2: an action is a short word made of numbers from 0 to 255.
The problem it solves: The same output format must work for a small robot transformer and for a large language model.
How it works: Both models tokenise the action into 256 bins uniformly distributed along each of eight dimensions: one for terminating the episode and seven for end-effector movement. RT-2 writes these as text tokens, for example “1 128 91 241 5 101 127”. A standard categorical cross-entropy loss is used for both: over the discrete buckets for RT-1 and over all possible language tokens for RT-2.
Example: Illustration, not from the paper The eight numbers of one step are the stop flag followed by x, y, z, roll, pitch, yaw and gripper.
Why the authors chose it: It lets the paper reuse RT-1 and RT-2 unchanged.
Technical details: OpenVLA later adopted this scheme with percentile bins; this paper uses uniform bins between the normalisation limits and does not say what range the limits cover. Its related work argues that the aim is not new architectures but to show that existing ones can use the data.
Evidence in the paper: §IV-B · §IV-C · Figure 3

Check your understanding: Why eight dimensions and not seven?

The extra dimension is for terminating the episode, so the policy can say that it is done.

4. RT-1-X: a small robot transformer

Intuition: A compact network built for control, fed with a history of images.
The problem it solves: A model for control needs to be fast enough to run locally at the robot’s rate.
How it works: RT-1 has 35 million parameters. It takes a history of 15 images; each image is processed by an ImageNet-pretrained EfficientNet and the instruction becomes a Universal Sentence Encoder embedding; the two are combined with FiLM (feature-wise linear modulation) layers, producing 81 vision-language tokens, which a decoder-only transformer turns into the tokenised action. RT-1-X is this network trained on the robotics mixture.
Example: Illustration, not from the paper At 3 to 10 Hz, the model runs on the robot’s own computer and answers in less than the control period.
Why the authors chose it: It is a strong, small model whose architecture is already known to work on one robot, so any difference from RT-1 comes from the data.
Technical details: RT-1-X is run locally at the robot’s rate; the small size is also why the paper reads its failure to beat RT-1 on the large datasets as underfitting.
Evidence in the paper: §IV-B · Figure 3 (left)

Check your understanding: Why does using the same architecture for RT-1 and RT-1-X matter?

It isolates the effect of the training data: any gain of RT-1-X over RT-1 on a given robot can be attributed to the mixture, not to a different network.

5. RT-2-X: a large vision-language-action model

Intuition: The big sibling: a web-trained vision-language model that also writes actions.
The problem it solves: A small network may not have the capacity to absorb data from many robots at once.
How it works: RT-2 is a family of vision-language-action (VLA) models trained on Internet-scale vision-language data together with robot control data, writing actions as text tokens. The paper uses the RT-2-PaLI-X variant, whose backbone is a vision transformer (ViT) and the UL2 language model, pretrained mostly on the WebLI dataset. RT-2-X is trained by co-fine-tuning with about one half web vision-language data and one half robotics mixture; the 55-billion-parameter model is hosted on a cloud service and queried over the network.
Example: Illustration, not from the paper A web image-question pair and a robot step with its action text are drawn in about equal numbers for each training batch.
Why the authors chose it: The authors expect, and find, that a high-capacity web-pretrained model can use the heterogeneous data where the small one underfits.
Technical details: A 5B-parameter variant is used for the ablations. The discussion offers the RT-1-X model as the foundation for future work; OpenVLA’s paper describes RT-2-X as closed-source and reachable only through an API.
Evidence in the paper: §IV-B · §IV-C · Table II

Check your understanding: What is “co-fine-tuning”?

Training on web data and robot data together, in this paper in roughly equal parts, so the model keeps what it learned from the web while learning to act.

Part C

Understand the data and the training

Three things to understand: what the dataset actually contains, which part of it the models were trained on, and what baselines they are compared with. The paper is short on training details, so this part also notes what is not said.

What is in the dataset (Figure 2)

Intuition: A big dataset is mostly a few big pieces and a long tail.
The problem it solves: The headline count hides how the data are distributed over robots, scenes and skills.
How it works: Figure 2 shows the breakdown of the 60 datasets by embodiment, with the Franka arm the most common, which is reflected in the largest number of distinct scenes (counted from dataset metadata). The xArm and the Google robot contribute the most trajectories, from a few large datasets. To analyse diversity the authors use the PaLM language model to extract objects and behaviours from the instructions: most skills are in the pick-and-place family, with a long tail such as wiping or assembling, and objects range from appliances to food and utensils. The dataset has 527 skills and 160,266 tasks according to the abstract.
Example: Illustration, not from the paper A policy trained on the pooled data meets many Franka scenes but few quadruped scenes, and many pick-and-place examples but few “assembling” ones.
Why the authors chose it: It shows what diversity the dataset has, along the axes the authors chose, and how it is concentrated.
Technical details: The figure uses pie and bar charts without printed numbers. The paper does not define a “skill” or a “task” in the counts (a later paper that lists the dataset reports 217 verbs and 311 scenes for it), and the language annotations are partial.
Evidence in the paper: §III-B · Figure 2 · Abstract

Check your understanding: Why do the Franka robot and the xArm dominate different charts?

The Franka has many small datasets, hence many scenes. The xArm and the Google robot have a few very large datasets, hence many trajectories.

The training mixture: 12 datasets, 9 robots

Intuition: The models were trained on the datasets available when the experiments ran, not on all 60.
The problem it solves: The dataset grew while the experiments were being done.
How it works: The robotics mixture used across all experiments is the data from 9 manipulators, taken from RT-1, QT-Opt, Bridge, Task Agnostic Robot Play, Jaco Play, Cable Routing, RoboTurk, NYU VINN, Austin VIOLA, Berkeley Autolab UR5, TOTO and Language Table. RT-1-X is trained on this mixture only; RT-2-X is co-fine-tuned with an approximately one-to-one split of web vision-language data and the mixture. All evaluations in Section V use the same mixture.
Example: Illustration, not from the paper A policy for the Berkeley UR5 is trained on the UR5 data together with data of 8 other robot types, not on all 22.
Why the authors chose it: The authors note the dataset has since been extended and plan to train on larger versions.
Technical details: The sampling weights of the 12 datasets, the number of training steps, the batch size and the hyperparameters are not given in this paper. Octo and OpenVLA, which train on 25 and 27 datasets from the collection, publish their mixtures.
Evidence in the paper: §IV-C · Section III

Check your understanding: The abstract says a model trained on “this data” shows positive transfer. Which data?

The 12-dataset, 9-embodiment mixture of Section IV-C, not the whole 60-dataset, 22-embodiment collection.

The baselines: the original method and RT-1 on its own

Intuition: Two yardsticks: the dataset authors’ own model, and the same network without the extra data.
The problem it solves: To claim that pooled data help, one must compare with the best that the data owners could already do.
How it works: For each evaluation domain there are two baselines. The Original Method is the model developed by the dataset’s creators and trained only on that dataset (for Bridge it is language-conditioned behaviour cloning, LCBC; the small domains use other methods named in the figure legend: MVP with a recurrent policy, a ResNet with an MLP, VINN, and TACOR or HULC2). The second is RT-1 trained on the dataset in isolation, which shows whether the RT-X architecture can represent several robots at once and whether co-training helps. All RT-X models in this section are co-trained on the full mixture.
Example: Illustration, not from the paper For NYU door opening the baseline is the nearest-neighbour method of the dataset’s paper, plus an RT-1 trained on the door data alone.
Why the authors chose it: The original methods have presumably been tuned for their data; RT-1 isolates the data effect.
Technical details: The baselines’ training budgets and tuning effort are not given. The Original Method differs by dataset, so the five “Original Method” bars are five different models.
Evidence in the paper: §V-A · Figure 4

Check your understanding: Why are there two baselines and not one?

Beating the original method shows the new model is competitive; beating the same architecture trained on one dataset shows the extra data are the reason.

Part D

Understand the experiments

The experiments answer three questions: does training on many robots improve performance on one robot, does it improve generalisation, and which design choices matter. This part explains each test and what the figures and tables say. Our re-calculations are in the folded examples.

The set-up: three questions, 3,600 trials, six robots

How it works

The authors ask (1) whether co-training on many robots’ data improves performance on the training tasks, (2) whether it improves generalisation to new, unseen tasks, and (3) how model size, architecture and dataset composition matter. They report a total of 3,600 evaluation trials across 6 robots. In-distribution tests use each dataset’s own robot and protocol: five small datasets (Kitchen Manipulation, Cable Routing, NYU Door Opening, AUTOLab UR5, Robot Play) and two large ones (Bridge on a WidowX at two sites, and the RT-1 data on the Google robot). The out-of-distribution tests use RT-2-X on the Google robot.

Technical details

Our calculation The paper does not give the trial count of any bar or row. Spread evenly over the roughly 40 printed result cells, 3,600 trials would be about 90 each, but this is our assumption. With 20 trials a bar has a standard error of 11 points, with 50 trials 7, with 90 trials 5.

Run the example: how far the headline gaps can be trusted

Our calculation

What each part does

Choose trial counts per bar: 20, 50 and 90.
Compute the standard error and the smallest gap an exact test treats as different.
Test the paper’s headline gaps at each count.

import math
# How far can the headline gaps be trusted? The paper gives 3,600 trials in total and no per-bar count, so we try several.
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
def se(p, n): return 100 * math.sqrt(p * (1 - p) / n)
print("trials per bar | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)")
for n in (10, 20, 50, 90, 200):
    base = n // 2
    gap = next((100 * (k - base) / n for k in range(base, n + 1) if fisher(k, n, base, n) < 0.05), None)
    print(f"{n:14d} | {se(0.5, n):9.1f} pts | {gap:.0f} points")
print()
print("headline gaps, exact test on counts if every bar had n trials")
cases = (("Figure 4, mean: RT-1-X 63 vs Original Method 41", 63, 41), ("Figure 4, mean: RT-1-X 63 vs RT-1 44", 63, 44), ("Figure 4, Cable Routing: 56 vs 24", 56, 24), ("Figure 4, Autolab UR5: RT-1-X 45 vs Original 53", 45, 53),
         ("Table I, Bridge IRIS: RT-2-X 50 vs RT-1 40", 50, 40), ("Table II: emergent skills RT-2-X 75.8 vs RT-2 27.3", 75.8, 27.3), ("Table II: 55B 75.8 vs 5B (no history) 14.5", 75.8, 14.5), ("Table II: with web pretraining 48.7 vs from scratch 0", 48.7, 0.0))
print(f"  {'':56s} " + "  ".join(f"n={n:<4d}" for n in (20, 50, 90)))
for name, a, b in cases:
    ps = [fisher(round(a * n / 100), n, round(b * n / 100), n) for n in (20, 50, 90)]
    print(f"  {name:56s} " + "  ".join(f"{p:6.3f}" for p in ps))
print()
print("conclusion: the large effects (cable routing, no web pretraining, 55B against 5B) hold at any plausible count; the small ones (a 10-point gap at one Bridge site, the UR5 reversal) need counts that the paper does not give")

Output:

trials per bar | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)
            10 |      15.8 pts | 50 points
            20 |      11.2 pts | 35 points
            50 |       7.1 pts | 22 points
            90 |       5.3 pts | 16 points
           200 |       3.5 pts | 10 points

headline gaps, exact test on counts if every bar had n trials
                                                           n=20    n=50    n=90  
  Figure 4, mean: RT-1-X 63 vs Original Method 41           0.205   0.027   0.004
  Figure 4, mean: RT-1-X 63 vs RT-1 44                      0.341   0.070   0.016
  Figure 4, Cable Routing: 56 vs 24                         0.105   0.002   0.000
  Figure 4, Autolab UR5: RT-1-X 45 vs Original 53           0.752   0.548   0.297
  Table I, Bridge IRIS: RT-2-X 50 vs RT-1 40                0.751   0.422   0.231
  Table II: emergent skills RT-2-X 75.8 vs RT-2 27.3        0.004   0.000   0.000
  Table II: 55B 75.8 vs 5B (no history) 14.5                0.000   0.000   0.000
  Table II: with web pretraining 48.7 vs from scratch 0     0.000   0.000   0.000

conclusion: the large effects (cable routing, no web pretraining, 55B against 5B) hold at any plausible count; the small ones (a 10-point gap at one Bridge site, the UR5 reversal) need counts that the paper does not give

Our calculation, assuming independent trials and one run; the counts are assumptions because the paper gives none.

Evidence in the paper

§V · Figure 4 · Table I

Check your understanding: What does “in-distribution” mean here?

The test uses the same robot and task type as the dataset it came from. It is not a test on a new robot or a new task; only the Google robot tests in Section V-B probe unseen objects and skills.

Small-data domains: RT-1-X against the original methods (Figure 4)

How it works

On five small datasets the RT-1-X model, co-trained on the mixture, beats the original method on 4 of 5 and beats RT-1 trained on the dataset alone on all 5. The mean over the five is 63% for RT-1-X, 41% for the original methods and 44% for RT-1. The authors read the large average gain as showing that domains with limited data benefit substantially from co-training, and call the gain 50% higher.

Technical details

Figure 4, success in percent, as printed on the bars.
Domain	Original method	RT-1 (dataset alone)	RT-1-X (mixture)
Kitchen Manipulation	43	48	63
Cable Routing	24	18	56
NYU Door Opening	53	65	80
Autolab UR5	53	25	45
Task-Agnostic Play	33	68	72
Mean	41	44	63

Our calculation The means reproduce (41.2, 44.8 and 63.2), with the RT-1 mean printed as 44 where rounding gives 45. “50% higher” is a relative figure: 22 points over the originals, which is 53%, and 18.4 points over RT-1, which is 41%. RT-1-X beats RT-1 in all 5 domains, which is the strongest possible sign-test result with 5 domains, p of 0.0625, and beats the originals in 4 (p of 0.375).

Run the example: means, rounding and sign tests

Our calculation

What each part does

Type in the bar values and recompute the means.
Convert the gains to points and to relative percentages.
Run sign tests, and recompute the Table I means.

import math
# Figure 4 (bar values) and Table I of the Open X-Embodiment paper: success in percent.
small = {  # domain: (Original Method, RT-1, RT-1-X)
 "Kitchen Manipulation": (43, 48, 63), "Cable Routing": (24, 18, 56), "NYU Door Opening": (53, 65, 80), "Autolab UR5": (53, 25, 45), "Task-Agnostic Play": (33, 68, 72)}
printed = (41, 44, 63)
print("Figure 4: means of the five domains, recomputed")
means = []
for j, name in enumerate(("Original Method", "RT-1", "RT-1-X")):
    m = sum(v[j] for v in small.values()) / 5
    means.append(m)
    print(f"  {name:16s} mean {m:5.2f}  rounded {round(m)}  truncated {int(m)}  printed {printed[j]}")
print(f"  RT-1-X over Original Method: +{means[2] - means[0]:.1f} points = {100 * (means[2] / means[0] - 1):.0f}% relative; over RT-1: +{means[2] - means[1]:.1f} points = {100 * (means[2] / means[1] - 1):.0f}% relative")
print("  the caption says 'RT-1-X mean success rate is 50% higher than that of either'")
print()
def sign_p(pos, neg):
    n = pos + neg; k = max(pos, neg)
    return min(1.0, 2 * sum(math.comb(n, i) for i in range(k, n + 1)) / 2 ** n)
for j, name in ((0, "Original Method"), (1, "RT-1")):
    w = sum(v[2] > v[j] for v in small.values()); l = sum(v[2] < v[j] for v in small.values())
    print(f"RT-1-X against {name:16s}: better in {w}, worse in {l} of 5 domains; sign test p = {sign_p(w, l):.4f}")
print("with five domains the smallest two-sided sign-test p that can occur is", f"{sign_p(5, 0):.4f}", "(5 wins out of 5)")
print("per domain, RT-1-X minus RT-1 (the same network, trained on one dataset or on the mixture):", [v[2] - v[1] for v in small.values()])
print()
print("Table I (large-dataset domains)")
tab1 = {"Bridge, IRIS (Stanford)": (13, 40, 27, 50), "Bridge, RAIL Lab (UCB)": (13, 30, 27, 30), "RT-1 paper, 6 skills": (None, 92, 73, 91)}
print(f"  {'setting':26s} {'Original':>9s} {'RT-1':>5s} {'RT-1-X':>7s} {'RT-2-X':>7s}")
for k, v in tab1.items():
    print(f"  {k:26s} {('-' if v[0] is None else str(v[0]) + '%'):>9s} {v[1]:>4d}% {v[2]:>6d}% {v[3]:>6d}%")
for j, n in ((1, "RT-1"), (2, "RT-1-X"), (3, "RT-2-X")):
    print(f"  mean of the three settings, {n:7s} {sum(v[j] for v in tab1.values()) / 3:5.1f}")
print("  RT-1-X is below RT-1 in all three settings (27 vs 40, 27 vs 30, 73 vs 92): the paper reads this as underfitting.")
print("  RT-2-X is 50 vs 40, 30 vs 30, 91 vs 92 against RT-1: ahead in one setting, equal in two. 'Outperforms both' rests on the Original Method row (13%) and one Bridge site.")

Output:

Figure 4: means of the five domains, recomputed
  Original Method  mean 41.20  rounded 41  truncated 41  printed 41
  RT-1             mean 44.80  rounded 45  truncated 44  printed 44
  RT-1-X           mean 63.20  rounded 63  truncated 63  printed 63
  RT-1-X over Original Method: +22.0 points = 53% relative; over RT-1: +18.4 points = 41% relative
  the caption says 'RT-1-X mean success rate is 50% higher than that of either'

RT-1-X against Original Method : better in 4, worse in 1 of 5 domains; sign test p = 0.3750
RT-1-X against RT-1            : better in 5, worse in 0 of 5 domains; sign test p = 0.0625
with five domains the smallest two-sided sign-test p that can occur is 0.0625 (5 wins out of 5)
per domain, RT-1-X minus RT-1 (the same network, trained on one dataset or on the mixture): [15, 38, 15, 20, 4]

Table I (large-dataset domains)
  setting                     Original  RT-1  RT-1-X  RT-2-X
  Bridge, IRIS (Stanford)          13%   40%     27%     50%
  Bridge, RAIL Lab (UCB)           13%   30%     27%     30%
  RT-1 paper, 6 skills               -   92%     73%     91%
  mean of the three settings, RT-1     54.0
  mean of the three settings, RT-1-X   42.3
  mean of the three settings, RT-2-X   57.0
  RT-1-X is below RT-1 in all three settings (27 vs 40, 27 vs 30, 73 vs 92): the paper reads this as underfitting.
  RT-2-X is 50 vs 40, 30 vs 30, 91 vs 92 against RT-1: ahead in one setting, equal in two. 'Outperforms both' rests on the Original Method row (13%) and one Bridge site.

Arithmetic on the printed bars.

Evidence in the paper

§V-A · Figure 4

Check your understanding: What can five domains show with a sign test?

Even a clean sweep of 5 wins out of 5 gives p of 0.0625, above the usual 0.05, so a significance claim has to rest on the size of the gaps and the trials in each bar, not on the number of domains.

Large-data domains: capacity matters (Table I)

How it works: On the two large datasets, Bridge and the RT-1 data, RT-1-X does not beat the RT-1 baseline trained on the dataset alone (27 against 40 and 30 at two Bridge sites, and 73 against 92 on the RT-1 tasks), which the authors read as underfitting by the small model. The 55-billion-parameter RT-2-X reaches 50, 30 and 91, outperforming the original method (13 at both Bridge sites) and about matching RT-1. They conclude that X-robot training can improve performance in data-rich domains only with a sufficiently high-capacity architecture.
Technical details: Our calculation Averaged over the three settings, RT-1 is 54.0, RT-1-X 42.3 and RT-2-X 57.0. Against RT-1, RT-2-X is ahead at one Bridge site (50 against 40), equal at the other (30 against 30) and 1 point behind on the RT-1 tasks (91 against 92); the gain over the original method is at the two Bridge sites only, where the original method scores 13. The table is titled a “parameter count scaling experiment”, but the two models differ in architecture, pretraining and training data as well as parameters.
Evidence in the paper: §V-A · Table I

Check your understanding: Does Table I show that more parameters help?

It shows that a large web-pretrained model with co-fine-tuning was not worse than RT-1 where the small model was. Architecture, web pretraining and the web data mixture also changed, so size alone is not isolated.

Generalisation and emergent skills (Table II, rows 1 to 3)

How it works

Two tests with RT-2-X. On unseen objects, backgrounds and environments, RT-2 and RT-2-X are on par (62% against 61%), which the authors expect since RT-2 already generalises through its vision-language backbone. On “emergent skills”, tasks with objects and skills that are in the Bridge data (a WidowX robot) but not in the Google robot’s RT-2 data, RT-2-X scores 75.8% against 27.3% for RT-2, about three times as much. Removing the Bridge data lowers it to 42.8%, which they take as evidence that the WidowX data are the source of the extra skills.

Technical details

Our calculation The ratio is 2.78. The removal of Bridge costs 33.0 points, 44% of the full model’s score. The exact trial counts are not given; the percentages cannot all come from one count (the smallest common count that fits all seven emergent-skill values is 322), and row (3) is the truncated 42.8 of a fraction such as 3/7.

Run the example: which trial counts fit the printed percentages

Our calculation

What each part does

Type in the seven emergent-skill percentages.
For each row, list the trial counts that could print it.
Look for a count that fits all rows.

import math
# The emergent-skills column of Table II is printed to one decimal. Which numbers of trials could produce it?
vals = {1: 27.3, 2: 75.8, 3: 42.8, 4: 44.4, 5: 14.5, 6: 0.0, 7: 48.7}
def fits(v, n):   # a count k out of n that prints as v, by rounding or by truncating to one decimal
    return [k for k in range(n + 1) if round(100 * k / n, 1) == v or math.floor(1000 * k / n) / 10 == v]
for r, v in vals.items():
    ns = [n for n in range(2, 81) if fits(v, n)]
    print(f"row ({r}) {v:5.1f}%: possible trial counts up to 80: {ns[:12]}{' ...' if len(ns) > 12 else ''}")
common = [n for n in range(1, 1000) if all(fits(v, n) for v in vals.values())]
print()
print("trial counts that fit all seven values at once (up to 999):", common[:6], "...")
print("rows (1) and (2) could share 33 trials: 9/33 and 25/33 print as", f"{100 * 9 / 33:.1f} and {100 * 25 / 33:.1f}")
print("row (3), 42.8, is 3/7 = 42.857 truncated, not rounded (rounded it would print 42.9); the other rows are rounded")
gen = [62, 61, 54, 52, 30, 1, 47]
print("the generalisation column (whole numbers) is less demanding; smallest common count up to 400:", [n for n in range(1, 400) if all(any(round(100 * k / n) == v for k in range(n + 1)) for v in gen)][:5])
print("so the emergent-skills rows were not all scored on the same number of trials, or the percentages are averages over unequal groups of tasks")

Output:

row (1)  27.3%: possible trial counts up to 80: [11, 22, 33, 44, 55, 66, 73, 77]
row (2)  75.8%: possible trial counts up to 80: [29, 33, 58, 62, 66]
row (3)  42.8%: possible trial counts up to 80: [7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77]
row (4)  44.4%: possible trial counts up to 80: [9, 18, 27, 36, 45, 54, 63, 72]
row (5)  14.5%: possible trial counts up to 80: [48, 55, 62, 69, 76]
row (6)   0.0%: possible trial counts up to 80: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] ...
row (7)  48.7%: possible trial counts up to 80: [39, 41, 76, 78, 80]

trial counts that fit all seven values at once (up to 999): [322, 380, 392, 414, 421, 425] ...
rows (1) and (2) could share 33 trials: 9/33 and 25/33 print as 27.3 and 75.8
row (3), 42.8, is 3/7 = 42.857 truncated, not rounded (rounded it would print 42.9); the other rows are rounded
the generalisation column (whole numbers) is less demanding; smallest common count up to 400: [79, 87, 89, 90, 92]
so the emergent-skills rows were not all scored on the same number of trials, or the percentages are averages over unequal groups of tasks

Arithmetic on the printed column; it shows which counts are possible, not which were used.

Evidence in the paper

§V-B · Figure 5 · Table II

Check your understanding: Does the Bridge-removal result prove that skills moved between robots?

It is consistent with it. Removing Bridge also removes a large share of the data and the objects that the test uses, so some of the drop could come from less data or from knowledge of the objects rather than the arm’s motions.

Design decisions: history, size, web pretraining (Table II, rows 4 to 7)

How it works

The ablations use mainly the 5-billion-parameter variant. A short image history raises emergent skills from 14.5% to 44.4%; web pretraining is critical (0% from scratch); the 55B model has a much higher emergent-skill success than the 5B (75.8% against 44.4%); and, unlike in the RT-2 paper, co-fine-tuning (44.4%) and plain fine-tuning (48.7%) are similar, which the authors attribute to the greater diversity of the robotics data.

Technical details

Table II, as printed (the success columns are percentages).
Row	Size	History	Data	Web co-training	Start	Emergent skills (%)	Generalisation (%)
(1) RT-2	55B	none	Google robot	yes	web-pretrained	27.3	62
(2) RT-2-X	55B	none	robotics data	yes	web-pretrained	75.8	61
(3)	55B	none	except Bridge	yes	web-pretrained	42.8	54
(4)	5B	2	robotics data	yes	web-pretrained	44.4	52
(5)	5B	none	robotics data	yes	web-pretrained	14.5	30
(6)	5B	2	robotics data	no	from scratch	0.0	1
(7)	5B	2	robotics data	no	web-pretrained	48.7	47

Our calculation Of the comparisons the text cites, only some change one factor. Rows (4) and (5) differ only in history, rows (4) and (7) only in web co-training, and rows (6) and (7) only in the starting checkpoint. The size comparison cited, rows (2) and (4), also differs in history; the clean size comparison is rows (2) and (5), 75.8 against 14.5 (5.2 times, not 1.7). “Web pretraining is critical” cites rows (4) and (6), which differ in both co-training and the starting checkpoint; the clean pair is (6) and (7).

Run the example: which comparisons change only one thing

Our calculation

What each part does

Type in the seven rows with their factors.
List all pairs of rows that differ in exactly one factor.
Compare with the pairs the text cites.

# Table II: which pairs of rows differ in exactly one factor? Percentages as printed.
rows = {  # row: (model, size, history, data, web co-training, initial checkpoint, emergent skills, generalisation)
 1: ("RT-2", "55B", "none", "Google Robot action", "Yes", "Web-pretrained", 27.3, 62), 2: ("RT-2-X", "55B", "none", "Robotics data", "Yes", "Web-pretrained", 75.8, 61),
 3: ("RT-2-X", "55B", "none", "Robotics data except Bridge", "Yes", "Web-pretrained", 42.8, 54), 4: ("RT-2-X", "5B", "2", "Robotics data", "Yes", "Web-pretrained", 44.4, 52),
 5: ("RT-2-X", "5B", "none", "Robotics data", "Yes", "Web-pretrained", 14.5, 30), 6: ("RT-2-X", "5B", "2", "Robotics data", "No", "From scratch", 0.0, 1),
 7: ("RT-2-X", "5B", "2", "Robotics data", "No", "Web-pretrained", 48.7, 47)}
names = ["model", "size", "history", "data", "web co-training", "initial checkpoint"]
print("pairs of rows that differ in exactly one factor (the model name RT-2 / RT-2-X is tied to the data, so it is not counted separately):")
for a in rows:
    for b in rows:
        if a < b:
            diff = [names[i] for i in range(1, 6) if rows[a][i] != rows[b][i]]
            if len(diff) == 1:
                print(f"  rows ({a}) vs ({b}): {diff[0]:19s} emergent skills {rows[a][6]:5.1f} -> {rows[b][6]:5.1f}   generalisation {rows[a][7]:3d} -> {rows[b][7]:3d}")
print()
print("the comparisons the text cites, and what they actually change:")
for a, b, claim in ((2, 4, "55B has higher emergent-skill success than 5B"), (4, 5, "history helps"), (4, 6, "web pretraining is critical"), (4, 7, "co-fine-tuning and fine-tuning are similar"), (1, 2, "RT-2-X outperforms RT-2 by about 3x")):
    diff = [names[i] for i in range(1, 6) if rows[a][i] != rows[b][i]]
    print(f"  ({a}) vs ({b}) '{claim}': differ in {', '.join(diff)}")
print()
print(f"ratio of rows (2) and (1): {75.8 / 27.3:.2f}; rows (2) and (5), the clean size comparison: {75.8 / 14.5:.2f}; rows (2) and (4), the cited one: {75.8 / 44.4:.2f}")
print(f"removing Bridge (row 3): {75.8 - 42.8:.1f} points lost, {100 * (75.8 - 42.8) / 75.8:.0f}% of the full model's score")
print("generalisation column: row (1) RT-2 62 vs row (2) RT-2-X 61 -> no gain, as the paper says")

Output:

pairs of rows that differ in exactly one factor (the model name RT-2 / RT-2-X is tied to the data, so it is not counted separately):
  rows (1) vs (2): data                emergent skills  27.3 ->  75.8   generalisation  62 ->  61
  rows (1) vs (3): data                emergent skills  27.3 ->  42.8   generalisation  62 ->  54
  rows (2) vs (3): data                emergent skills  75.8 ->  42.8   generalisation  61 ->  54
  rows (2) vs (5): size                emergent skills  75.8 ->  14.5   generalisation  61 ->  30
  rows (4) vs (5): history             emergent skills  44.4 ->  14.5   generalisation  52 ->  30
  rows (4) vs (7): web co-training     emergent skills  44.4 ->  48.7   generalisation  52 ->  47
  rows (6) vs (7): initial checkpoint  emergent skills   0.0 ->  48.7   generalisation   1 ->  47

the comparisons the text cites, and what they actually change:
  (2) vs (4) '55B has higher emergent-skill success than 5B': differ in size, history
  (4) vs (5) 'history helps': differ in history
  (4) vs (6) 'web pretraining is critical': differ in web co-training, initial checkpoint
  (4) vs (7) 'co-fine-tuning and fine-tuning are similar': differ in web co-training
  (1) vs (2) 'RT-2-X outperforms RT-2 by about 3x': differ in data

ratio of rows (2) and (1): 2.78; rows (2) and (5), the clean size comparison: 5.23; rows (2) and (4), the cited one: 1.71
removing Bridge (row 3): 33.0 points lost, 44% of the full model's score
generalisation column: row (1) RT-2 62 vs row (2) RT-2-X 61 -> no gain, as the paper says

Arithmetic on the printed table; it counts factors, not trials.

Evidence in the paper

§V-C · Table II

Check your understanding: Which Table II comparison is the cleanest test of web pretraining?

Rows (6) and (7): both are not co-trained and differ only in starting from scratch or from a web-pretrained model (0.0% against 48.7%).

What the results show

A model trained on pooled data from several robots does better than the dataset authors’ own methods on 4 of 5 small datasets, and RT-1-X beats RT-1 trained on a dataset alone in all 5.
A large web-pretrained model can use the pooled data even on robots that already have much data of their own, and gains skills that only other robots’ data contained, according to the paper’s tests.
Web pretraining is essential for the large model, and an image history helps.

What they do not show

Transfer to a robot that is not in the training data: all tested robots were in the mixture.
That the gaps are more than chance: no trial counts or variances are given, and the Table II percentages cannot share one count.
That model size is the reason RT-2-X beats RT-1-X on the large datasets: architecture, pretraining and data change together.
When pooled data help and when they hurt: the paper lists this as future work, and the Autolab UR5 result went the other way (45 against 53).

Part E · optional

Engineering deep dive

The data format, the action tokens, the two architectures, the training mixture and the evaluation settings, for readers who want to use or audit the work. Everything is collapsed; Parts A to D do not depend on it.

The data format and the repository (Section III)

Reported by the paper The format is RLDS (reinforcement learning datasets): serialised tfrecord files with episodes of steps, accommodating different numbers of RGB cameras, depth cameras and point clouds, with parallelised loading in the major deep learning frameworks.
Reported by the paper The repository offers the dataset (1M+ trajectories, 22 embodiments), a selection of pre-trained RT-X checkpoints for inference and finetuning, and open-source tools; a table of all 60 datasets is on the project website, not in the paper.

The action tokens

Reported by the paper Each model outputs eight numbers per step: a termination flag and seven end-effector dimensions. Each is cut into 256 bins uniformly spaced between the dataset’s normalisation limits. For RT-2 the tokens are written as text, for example “1 128 91 241 5 101 127” (seven numbers as in the paper’s example). The loss is the categorical cross-entropy over the discrete buckets (RT-1) or over all language tokens (RT-2). Our rendering of the binning:

b=\frac{\lfloor 256\cdot (a-a_{\min})}{(a_{\max}-a_{\min})\,\rfloor },\,a=a_{\min}+\frac{(b+\tfrac{1}{2})\cdot (a_{\max}-a_{\min})}{256}

The formula is ours; the paper gives no equation and does not state the limits a_min and a_max.

The two architectures (Figure 3)

As printed in Section IV and Figure 3.
	RT-1-X	RT-2-X
Size	35 million parameters	55 billion (and 5 billion for ablations)
Vision	ImageNet-pretrained EfficientNet, history of 15 images	vision transformer (ViT) of the PaLI-X model
Language	Universal Sentence Encoder embedding, FiLM-conditioned	UL2 language model, pretrained on WebLI
Output	decoder-only transformer, 256 bins x 8 dimensions	action text tokens, 256 bins x 8 dimensions
Training data	robotics mixture only	co-fine-tuned: about half web vision-language data, half robotics mixture
Inference	local, 3 to 10 Hz	cloud-hosted, queried over the network, 3 to 10 Hz

The training mixture (Section IV-C)

Reported by the paper Data from 9 manipulators: RT-1, QT-Opt, Bridge, Task Agnostic Robot Play, Jaco Play, Cable Routing, RoboTurk, NYU VINN, Austin VIOLA, Berkeley Autolab UR5, TOTO and Language Table (12 datasets in our count). The mixture is all the data available when the experiments were run; the entire collection has 22 embodiments. Weights, steps and hyperparameters are not stated.

The evaluation settings (Section V and Table I)

Assembled from Figure 4, Table I and the text of Section V-A; robot and original-method assignments for the small domains are our reading of the figure and may not match the paper’s own labels. The paper does not list trial counts.
Setting	Robot	Where	Original method
Kitchen Manipulation	Jaco	—	MVP with a recurrent policy (BC-RNN)
Cable Routing	Franka (per the dataset)	—	ResNet with an MLP
NYU Door Opening	Hello Stretch	—	VINN
Autolab UR5	UR5	—	ResNet with an MLP
Robot Play	Franka (per the dataset)	—	TACOR or HULC2 (as printed in the figure legend)
Bridge	WidowX	IRIS (Stanford)	LCBC (language-conditioned behaviour cloning)
Bridge	WidowX	RAIL Lab (UC Berkeley)	LCBC
RT-1 paper, 6 skills	Google robot	Google robotics lab	—

Limitations stated by the authors (Section VI)

The experiments do not consider robots with very different sensing and actuation modalities.
They do not study generalisation to new robots.
They do not provide a decision criterion for when positive transfer does or does not happen.

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: where its numbers and counts disagree, where a comparison changes more than one thing, and where missing trial counts limit the conclusions. Open X-Embodiment is a landmark data release that later models rely on, and its contribution does not depend on the details below; the findings concern the strength of the evidence in the paper’s experiments, which are short (six pages) and sparsely documented. Each one links back to the concept it affects, and the calculations are reproducible in the examples.

Numbers that do not agree

F1. Counts that are not defined or that vary: institutions, labs, skills, tasks Numerical inconsistency

What the paper says. The dataset has 1M+ trajectories from 22 embodiments, pooled from 60 datasets from 34 labs, through a collaboration of 21 institutions, demonstrating 527 skills (160,266 tasks).

What we found. The text gives both 21 institutions and 34 labs, which fit only if labs are counted inside institutions; the paper does not say so. “527 skills (160,266 tasks)” has no definition of either unit: 160,266 tasks over 527 skills is 304 tasks per skill, and the language annotations used to count them are partial. The trajectory count has moved from “1M+” here to 1.4 million and more than 2 million in later papers’ descriptions of the growing collection, and the same dataset is cited with 311 scenes and 217 verbs by another paper but with neither number here.

Why it matters. The headline counts are reasonable but cannot be reproduced from the paper, and they change with versions of the dataset, so anyone quoting them should say which version. Concept: The repository · What is in the dataset · example
F2. “50% higher” is a relative figure, and the RT-1 mean is truncated Numerical inconsistency

What the paper says. RT-1-X mean success rate is 50% higher than that of either the Original Method or RT-1.

What we found. The bar means are 41.2, 44.8 and 63.2 (printed 41, 44 and 63, so the RT-1 mean is truncated where rounding gives 45). RT-1-X is 22.0 points ahead of the original methods, 53% in relative terms, and 18.4 points ahead of RT-1, 41% in relative terms; “50%” is therefore neither a gain in points nor exact for both baselines.

Why it matters. A small imprecision, but “50% higher” can be read as 50 percentage points, which would be more than double the actual gap. Concept: Small-data domains · example
F3. The printed percentages cannot all come from one trial count Numerical inconsistency

What the paper says. The paper reports 3,600 evaluation trials in total across 6 robots, and gives success percentages to one decimal in Table II.

What we found. No bar, row or setting has a stated trial count. The seven emergent-skills percentages of Table II (27.3, 75.8, 42.8, 44.4, 14.5, 0.0, 48.7) need different counts: 75.8 fits 29, 33, 58, 62 or 66 trials, 14.5 fits 48, 55 or 62, 48.7 fits 39 or 41, 44.4 fits multiples of 9, and the smallest count that fits all seven is 322. The 42.8 is a truncation of a fraction such as 3/7, where the other values are rounded. The rows were therefore not all scored on the same trials, or are averages over unequal groups of tasks.

Why it matters. Differences between rows may reflect different test sets as well as different models, and no interval can be computed without counts. Concept: Emergent skills · The set-up · example
F4. The abstract’s data (22 robots) and the training data (9 robots) differ Numerical inconsistency

What the paper says. We assemble a dataset from 22 different robots… a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer.

What we found. Section IV-C says the experiments’ mixture covers 9 embodiments, 12 named datasets of the 60 (20%) and 9 of the 22 embodiments (41%), because the dataset grew after the experiments were set up. The 22-robot collection and the 9-robot training data are both called the data in different places.

Why it matters. The positive transfer was shown for a 9-robot subset; nothing in the paper tests what the full 22-robot collection does. Concept: The training mixture · example

How far the claims go

F5. The “parameter count scaling experiment” changes more than parameters Method caveat

What the paper says. Table I: a parameter count scaling experiment to assess the impact of capacity on absorbing large-scale diverse embodiment data; RT-1-X underfits, and RT-2-X with significantly more parameters obtains strong performance.

What we found. RT-1-X (35 million parameters, trained from scratch on robot data) and RT-2-X (55 billion, a web-pretrained vision-language model co-fine-tuned with web data) differ in architecture, pretraining and training data, so there are only two sizes and they are confounded with everything else. No training loss is shown, so “underfits” is an inference from success rates. Against RT-1 the 55-billion-parameter model is ahead at one Bridge site (50 against 40), equal at another (30) and 1 point behind on the RT-1 tasks (91 against 92); the clear gain is over the original method (13) at the two Bridge sites.

Why it matters. The result supports “a large web-pretrained model works where the small one does not” but not a conclusion about capacity alone. Concept: Large-data domains · example
F6. Positive transfer rests on five domains, unknown trial counts and one reversal Method caveat

What the paper says. RT-1-X outperforms the original method on 4 of 5 datasets, with a large average improvement, showing that limited-data domains benefit substantially from co-training on X-embodiment data.

What we found. RT-1-X beats the original method in 4 of 5 domains and RT-1 in 5 of 5; with five domains the best possible sign test is p of 0.0625. The Autolab UR5 result reverses (45 against 53 for the original), and the baselines are inconsistent: RT-1 alone is below the original method on the UR5 (25) and Cable Routing (18) data and above it elsewhere. Without trial counts, a gap of 20 points between two bars is within chance at 20 trials per bar (exact p of 0.21 and 0.34 for the two mean gaps) and clear at 90. The large Cable Routing gain (56 against 24) holds at any plausible count.

Why it matters. The direction is probably right and the largest gains are convincing; the average hides how uneven the gains are across domains and how small they could be. Concept: Small-data domains · The baselines · example · example
F7. “Emergent skills from other robots” cannot be told from other explanations Method caveat

What the paper says. RT-2-X outperforms RT-2 by about 3x on skills that are not in the Google robot’s data but are in the WidowX data; removing the Bridge data reduces performance, suggesting transfer from the WidowX data.

What we found. Rows (1) and (2) differ in the data (the Google robot’s alone against the pooled mixture), which also adds objects, language and scenes; removing Bridge (row 3) removes 33.0 points, 44% of the score, but also removes a large dataset and the very objects the test uses. There is no control that trains on the Bridge data alone with a Google-robot dataset, or on Bridge language and objects without the arm’s motions. The trial counts are unknown and the percentages cannot share a count (F3).

Why it matters. The result shows that adding other robots’ data helps on tasks the data contain; “transfer of motor skills across embodiments” is one possible reading, not the only one. Concept: Emergent skills · example
F8. Several of the cited ablation comparisons change two things Method caveat

What the paper says. The 55B model has a significantly higher emergent-skills success rate than the 5B (rows 2 and 4); web pretraining is critical (rows 4 and 6).

What we found. Rows (2) and (4) differ in size and in history; rows (4) and (6) differ in web co-training and in the starting checkpoint. The single-factor pairs are rows (2) and (5) for size (75.8 against 14.5, 5.2 times, where the cited pair gives 1.7 times), rows (6) and (7) for web pretraining (0.0 against 48.7), (4) and (5) for history and (4) and (7) for web co-training. The conclusions survive, and the size effect is larger than the cited comparison suggests.

Why it matters. The ablation table is informative but needs to be read pair by pair; the clean pairs support the paper’s conclusions more strongly than the cited ones. Concept: Design decisions · example

What is not tested

F9. No test on a robot that was not in the training data Limitation

What the paper says. Can we instead train a generalist X-robot policy that can be adapted efficiently to new robots, tasks and environments?

What we found. Every evaluation robot (WidowX, Google robot, Jaco, the UR5 and the others) has its own dataset in the training mixture. The limitations section concedes that generalisation to new robots is not studied. Later papers (Octo, OpenVLA) test finetuning to new robots, with the pretrained model as a starting point.

Why it matters. The opening question of the abstract is only half answered: the paper shows that pooled data help known robots. Concept: The set-up · Coarse alignment
F10. Training and evaluation details that are missing Limitation

What the paper says. The paper provides data, models and tools, and aims to energise research around X-embodiment robotic learning.

What we found. The sampling weights of the 12 training datasets, the number of steps, batch sizes and hyperparameters are not given, nor the trial count or variance of any result, nor the budget spent tuning the original-method baselines. The dataset table of all 60 datasets is on a website. The 55-billion-parameter RT-2-X is served through a cloud service; the discussion offers RT-1-X as the foundation model, and OpenVLA’s paper calls RT-2-X closed-source.

Why it matters. The dataset is fully usable; the RT-X results cannot be reproduced from the paper, and later work reports its own mixtures and trial counts. Concept: The training mixture · RT-2-X
F11. “No mechanism for the embodiment gap” is a design choice with open consequences Ambiguity

What the paper says. We directly train a policy on X-embodiment data, without any mechanisms to reduce the embodiment gap, and observe positive transfer.

What we found. The paper says the same action vector may induce different motions on different robots and the camera views differ widely, and that outputs are de-normalised per embodiment. It does not say how the model knows which robot it controls, whether the embodiment or dataset is an input, or how often it confused robots. Later models add per-robot encoders or heads (Octo, GR00T N1).

Why it matters. The positive-transfer result is evidence that the data help despite the gap; it does not show that ignoring the gap is the best approach, and the paper leaves when transfer fails for future work. Concept: Coarse alignment · Small-data domains

Knowledge check

Try each question first, then reveal the explanation.

Question 1

In your own words: how does Open X-Embodiment make many robots’ data usable together?

Explanation

It converts 60 datasets to one file format, keeps one camera view per dataset, turns actions into a common 7-number hand action plus a stop flag, normalises each dataset on its own and cuts each number into 256 bins. The meaning of a token still depends on the robot.

Question 2

What do the RT-1-X experiments show on small and large datasets?

Explanation

On five small datasets RT-1-X trained on the pooled data beats the original method on four and RT-1 on all five (mean 63 against 41 and 44). On the two large datasets the small model does not beat RT-1; the 55-billion-parameter RT-2-X does better.

Question 3

Which Table II comparison is a clean test of web pretraining, and which is not?

Explanation

Rows (6) and (7) differ only in the starting checkpoint (0.0 against 48.7). Rows (4) and (6), which the text cites, also differ in whether web data are co-trained.

Question 4

Name one thing the paper does not tell you.

Explanation

For example: the trial count of any bar, the sampling weights of the training mixture, how the model knows which robot it controls, or how it performs on a robot that is not in the data.

Glossary

Embodiment: A robot platform: its arms, grippers, cameras and action format.
X-embodiment training: Training one policy on data from many different robots.
Open X-Embodiment (OXE): The pooled collection of robot datasets and the open repository around it.
RLDS: Reinforcement learning datasets: a file format for episodes of observations and actions, stored as tfrecord files.
Trajectory: One recorded episode of a robot doing a task.
Original Method: The model that the dataset’s creators trained only on their own dataset; a baseline.
RT-1: A 35-million-parameter transformer policy for robot control.
RT-1-X: RT-1 trained on the pooled robotics mixture.
RT-2: A vision-language model finetuned to write robot actions as text tokens.
RT-2-X: RT-2 co-fine-tuned on web data and the pooled robotics mixture; 55 billion parameters.
Co-fine-tuning: Training on web and robot data together so the model keeps web knowledge.
FiLM: Feature-wise linear modulation: a way to condition a network’s features on another input such as a sentence.
Action token: One of 256 bins of one action dimension, written as a token.
Positive transfer: Doing better on one robot because of data from other robots.
Emergent skills: Skills in the test that are not in the robot’s own data but are in other robots’ data.
In-distribution: Tested in the conditions of a dataset’s own robot and tasks.
Underfitting: A model too small to fit all its training data well.
Sign test: A test that counts in how many settings one method beats another, ignoring by how much.
LCBC: Language-conditioned behaviour cloning: the original method for the Bridge data.

References

Numbered notes in the text, like ^[1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

Open X-Embodiment Collaboration (2023). Open X-Embodiment: Robotic learning datasets and RT-X models. arXiv:2310.08864
Cited for:
- Pools more than one million robot trajectories from 22 robot embodiments, assembled by 21 institutions and covering 527 skills. Checked against the paper’s abstract.
- RT-1-X (35 million parameters, a FiLM-conditioned EfficientNet and a transformer) and RT-2-X (55 billion parameters, a PaLI-X vision-language model written to output action text) are trained on data from 9 manipulators, with actions cut into 256 bins in eight dimensions; RT-2-X is co-fine-tuned on about equal parts of web vision-language data and the robotics mixture. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
- A high-capacity model (RT-X) trained on the pooled data shows positive transfer, improving several robots using experience from other platforms. Checked against the paper’s abstract.

Go further

All paper guides · Back to the VLA track

How to use this page

Start here: the paper at a glance

The problem

The main contribution

How the data and the models work, in one paragraph

What it learns from, and what it is tested on

What the paper reports

What to keep in mind

Understand the method

1. The repository: 60 datasets in one format

2. Coarse alignment of inputs and actions

3. Actions as 256-bin tokens, eight dimensions

4. RT-1-X: a small robot transformer

5. RT-2-X: a large vision-language-action model

Understand the data and the training

What is in the dataset (Figure 2)

The training mixture: 12 datasets, 9 robots

The baselines: the original method and RT-1 on its own

Understand the experiments

The set-up: three questions, 3,600 trials, six robots

Small-data domains: RT-1-X against the original methods (Figure 4)

Large-data domains: capacity matters (Table I)

Generalisation and emergent skills (Table II, rows 1 to 3)

Design decisions: history, size, web pretraining (Table II, rows 4 to 7)

What the results show

What they do not show

Engineering deep dive

Critical paper review

Numbers that do not agree

F1. Counts that are not defined or that vary: institutions, labs, skills, tasks Numerical inconsistency

F2. “50% higher” is a relative figure, and the RT-1 mean is truncated Numerical inconsistency

F3. The printed percentages cannot all come from one trial count Numerical inconsistency

F4. The abstract’s data (22 robots) and the training data (9 robots) differ Numerical inconsistency

How far the claims go

F5. The “parameter count scaling experiment” changes more than parameters Method caveat

F6. Positive transfer rests on five domains, unknown trial counts and one reversal Method caveat

F7. “Emergent skills from other robots” cannot be told from other explanations Method caveat

F8. Several of the cited ablation comparisons change two things Method caveat

What is not tested

F9. No test on a robot that was not in the training data Limitation

F10. Training and evaluation details that are missing Limitation

F11. “No mechanism for the embodiment gap” is a design choice with open consequences Ambiguity

Knowledge check

Question 1

Question 2

Question 3

Question 4

Glossary

References

Go further