5.3 — Open X-Embodiment

Learning objectives

After this lesson, you should be able to:

• State the scale of Open X-Embodiment.
• Explain why datasets must be harmonised before pooling.
• Describe positive transfer between robots.
• Explain the risk of unharmonised data.

The one-sentence idea

Many small datasets, made comparable, add up to one large one.

Pooling only helps if the data mean the same thing.

One dataset from many labs

Every robot lab used to collect its own data in its own format for its own robot. Open X-Embodiment was a collaboration to pool them. The released data contain more than a million robot trajectories from 22 robot embodiments, assembled by 21 institutions and covering 527 skills.^[1] The authors trained a high-capacity model, RT-X, on the pooled data and report positive transfer: it improves several robots using experience from other platforms.^[1] For the paper’s tables, a re-check of its numbers and a critical review, see the Open X-Embodiment paper guide^[1]; for a dataset that is deliberately single-robot but spread over 564 scenes, see the DROID paper guide.^[2]

That is the argument for “generalist” robot policies: a robot with little data of its own could borrow from others. The hard part is making very different datasets mean the same thing.

Making datasets speak the same language

Imagine two labs. Lab A records the hand position in metres at 5 Hz; lab B records it in millimetres at 10 Hz. Before pooling, you must convert units and bring both to a common rate. RLDS exists for exactly this kind of work: a toolkit for recording, replaying, editing and sharing data for sequential decision making, built around a standardised, lossless format.^[3] Here is the conversion, step by step:

import numpy as np

# Lab A: metres, 5 steps per second.  Lab B: millimetres, 10 steps per second.
a_times = np.arange(0, 2.0, 1/5);  a_pos = 0.10 * a_times            # moving at 0.10 m/s
b_times = np.arange(0, 2.0, 1/10); b_pos_mm = 150 * b_times           # moving at 150 mm/s

# common format: metres, 5 steps per second
b_pos = b_pos_mm / 1000.0                                             # convert units
b_pos_5hz = np.interp(a_times, b_times, b_pos)                        # resample to lab A's times

print("lab A speed (m/s):", round(float(np.diff(a_pos).mean() / (1/5)), 3))
print("lab B speed, raw numbers per step:", round(float(np.diff(b_pos_mm).mean()), 1), "(mm per step, 10 Hz)")
print("lab B speed after conversion (m/s):", round(float(np.diff(b_pos_5hz).mean() / (1/5)), 3))

lab A speed (m/s): 0.1
lab B speed, raw numbers per step: 15.0 (mm per step, 10 Hz)
lab B speed after conversion (m/s): 0.15

Toy model. Two imaginary labs and straight-line motions. Real harmonisation also has to align camera views, action definitions (lesson 4.1) and instructions.

Without the conversion, one of lab B’s steps is the number 15 where lab A’s equivalent step is 0.02: a model sees values hundreds of times too large. The same trap as the wrong statistics in lesson 4.3.

Why pooling can help, and how it can hurt

Suppose robot A has only 4 demonstrations of a reaching task, while robot B has 500 of the same task. If their data are made comparable, B’s data teach A’s model the common structure. If they are not, B’s data are confusing noise:

import numpy as np
rng = np.random.default_rng(0)

def demos(n, unit):                                               # action = 0.3 * (goal - hand), recorded in some unit
    X = rng.uniform(-1, 1, (n, 2))                                # goal minus hand
    Y = 0.3 * X * unit + rng.normal(0, 0.1 * unit, (n, 2))
    return X, Y

def fit(X, Y):  return np.linalg.lstsq(X, Y, rcond=None)[0]

def test_error(W):
    X, Y = demos(2000, 1.0)
    return float(np.mean((X @ W - Y) ** 2))

XA, YA = demos(4, 1.0)                                            # robot A, only four demonstrations
XB, YB = demos(500, 1.0)                                          # robot B, same units after harmonising
XC, YC = demos(500, 1000.0)                                       # robot B's data in millimetres, not converted

print("A alone (4 demos):               ", round(test_error(fit(XA, YA)), 5))
print("A + B, harmonised (504 demos):    ", round(test_error(fit(np.vstack([XA, XB]), np.vstack([YA, YB]))), 5))
print("A + B, raw millimetres (504 demos):", round(test_error(fit(np.vstack([XA, XC]), np.vstack([YA, YC]))), 1))

A alone (4 demos):                0.01747
A + B, harmonised (504 demos):     0.00979
A + B, raw millimetres (504 demos): 30458.5

Toy model. A shared linear rule and a unit mismatch. The lesson is qualitative: shared structure lets small datasets borrow strength, but only when the data mean the same thing.

Harmonised pooling cuts the error from 0.017 to 0.0098, which is about the noise level of the data themselves (0.1² = 0.01); unharmonised pooling destroys the model. The Open X-Embodiment authors report the positive case for their data;^[1] but each result is for particular robots and tasks, and it is not a guarantee for any robot you may have.

Why it matters for VLAs

Large VLAs need large data. OpenVLA was trained on 970,000 real-world robot demonstrations,^[4] and part of what makes such numbers reachable is that datasets like this exist. The X-embodiment idea also raises a question that returns in lesson 11.3: how a single policy should handle robots with different bodies.

Common misconceptions

Engineering takeaways

1. 1M+ trajectories, 22 embodiments, 21 institutions.
2. Harmonise units, rates and formats.
3. Positive transfer was reported.
4. Mismatched data can hurt.

Same words, same units, then pool.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: how web-scale pretraining and robot demonstrations combine in training.

Back to the VLA roadmap