VLA · Paper guide

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

The dataset that spreads one robot across many rooms, explained through the questions an engineer would ask before training on it.

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna et al.
arXiv:2403.12945 (v2)
Overview: about 4 minutes
Full guide: about 45 minutes

This is a guide to the DROID paper, not a sentence-by-sentence paraphrase. DROID is a dataset paper, so the “method” is how the data are collected and labelled and the “experiments” ask whether the data help a policy. It is the data counterpart to the model guides: OpenVLA and Octo both train on pooled robot datasets, and DROID is one of the datasets in OpenVLA’s mixture. Part A is a 4-minute overview. Parts B to D explain the collection, the data and the experiments one idea at a time. Tables and calibration details are folded away in Part E, and Part F is a critical review of the paper’s numbers and claims.

Start with the overview Jump to the critical review

How to use this page

In a hurry: read Part A (about 4 minutes).
To understand the dataset: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example, then technical details and where to find them in the paper.
To check or use it: open the folded sections (the comparison table, recorded fields, calibration and runnable examples) and read Part F.

Open the paper (arXiv:2403.12945) beside this page; section, table and figure references are to it. We paraphrase and never reproduce its text. Statements are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own small model of an idea, and are never presented as results of the paper.

Part A · about 4 minutes

Start here: the paper at a glance

The problem

A robot policy is only as general as the data it learns from, and robot demonstrations cannot be scraped from the internet: someone has to drive a robot, record what it sees and does, and repeat this in many places. As a result the biggest robot datasets of the time were collected in a few labs, in front of a few tables, with a handful of cameras. The authors ask for the missing ingredient: demonstrations from many different rooms, buildings, tasks and camera angles, all recorded on the same robot so that anyone can use them.

The main contribution

A dataset, DROID (Distributed Robot Interaction Dataset). 76 thousand successful demonstrations, about 350 hours, in 564 scenes in 52 buildings, with 86 distinct task verbs, collected by 50 people at 13 institutions over 12 months. Each episode has three synchronised camera streams, camera calibration, depth and one to three natural-language instructions.^[1]
A shared, reproducible way to collect it. The same portable robot station (a Franka arm on a height-adjustable desk with two moveable external cameras and a wrist camera) was built 18 times, and a collection protocol makes collectors move scenes about every 20 minutes and follow randomly sampled tasks.^[1]
A test of whether the diversity helps. Six real-robot tasks in four places, each with an out-of-distribution (OOD) variant, trained with a standard diffusion policy and compared with training on the task data alone and with adding the Open X-Embodiment collection.^[1]

How the dataset and the test work, in one paragraph

A collector wheels the station to a new spot, places the two external cameras, calibrates them with a checkerboard, and types the tasks that are possible in this scene into a small interface. The interface then draws one task at random for each demonstration and, from time to time, asks the collector to change the scene: nudge the base, move the cameras, change the lights, add or remove objects. The collector steers the arm with a virtual-reality headset’s controllers; success is marked by hand; crowd workers write the instructions afterwards. To test the data, the authors train a diffusion policy for each task on 50 to 150 task demonstrations, with half of every training batch coming from the extra dataset (DROID, Open X-Embodiment, or nothing), and run 10 rollouts per task setting.

What it learns from, and what it is tested on

The data: 76 thousand successful and about 16 thousand unsuccessful episodes recorded at 15 Hz on one robot type. In the experiments only the first 40 thousand successful, annotated episodes were used. Tested on: six tasks, from closing a waffle maker to cooking lentils, in a lab, an office and a home kitchen, each in the original setting and with distractors, new objects or a camera shift, always on the same hardware stack as the dataset.

What the paper reports

Figures 8 and 10 and Table I, as printed.
Test	DROID co-training	Comparisons
In distribution, 6 tasks (success, %)	75	53 no co-training; 43 Open X-Embodiment
Out of distribution, 7 settings (%)	53	17 no co-training; 36 Open X-Embodiment
Scene diversity, 3 OOD tasks, 7,362 trajectories (%)	60 from random scenes	40 from the 20 scenes with the most data
Scale of the dataset	76 thousand episodes, 564 scenes, 86 verbs	Open X-Embodiment: 1.4 million episodes, 311 scenes, 217 verbs

What to keep in mind

The experiments have 10 rollouts per task setting and one run each; the gap to Open X-Embodiment out of distribution (17 points) is not clearly beyond chance, and the gap between scene subsets (20 points) is not either.
The policies are trained and tested on the same robot, cameras and controller as DROID, while Open X-Embodiment mixes many robots, so “diverse data helps” is partly “data from the right robot helps”.
“Scenes” and “tasks” are the authors’ own counts: a task is a verb, a scene a workspace, and several of the printed counts (institutions, scene types, calibrated scenes) disagree with each other.
DROID is a dataset, not a model; it is a training ingredient, and the paper tests a single policy class.

If you read nothing else: DROID is a carefully built, openly released dataset that is far more varied in scenes and camera angles than earlier single-robot collections, and co-training with it clearly helps a small policy on six tasks; the paper shows that diverse in-embodiment data help but does not separate scene diversity from matching the robot, and the experiments are too small to rank the data sources finely.

Check your understanding: Why does the DROID collection protocol ask for tasks to be drawn at random, and for scenes to change every 20 minutes?

Without it collectors tend to repeat easy tasks near the robot in one place. Random prompts give coverage of tasks the collector would not have chosen, and frequent scene changes spread the 76 thousand episodes over many workspaces instead of a few.

Part B

Understand the method

For a dataset paper the “method” is the collection system: the robot station, the protocol that makes collectors produce varied data, and the labels added afterwards. Each idea below follows the same pattern, from intuition to where to find it in the paper.

One demonstration. Orange marks the design choices the guide calls out as the paper’s main contribution. Drawn by us from Section III, Figure 2 and Appendix F of the paper.

1. One robot station, built many times

Intuition: If every site uses the same machine, the data can be pooled without translation.
The problem it solves: Collecting data in many places usually means many different robots, cameras and controllers, which is why pooled datasets mix formats and are hard to use.
How it works: The station is a Franka Emika Panda 7-joint arm with a Robotiq two-finger gripper, mounted on a height-adjustable desk with wheels. Two Zed 2 stereo cameras on adjustable tripods give the “external” views, and a Zed Mini stereo camera on the wrist gives a third. The collector teleoperates with the controllers of a Meta Quest 2 virtual-reality headset, which set the arm’s 6-dimensional pose and the gripper opening. A control laptop runs the collection interface, a small computer hosts the Polymetis controller, and everything runs on one power cable. It was built 18 times in North America, Asia and Europe, and the hardware and software guide is public.
Example: Illustration, not from the paper A new lab orders the parts, follows the build guide, and its first recorded episode has the same cameras, gripper and 15 Hz control as an episode from another continent.
Why the authors chose it: It lets 50 collectors at 13 institutions produce one coherent dataset, and lets later users evaluate a policy on the same hardware.
Technical details: Observations are recorded at 15 Hz: three stereo RGB streams at 1280 × 720 resolution, joint positions and velocities (7 numbers), the end-effector pose and velocity (6 numbers) and the gripper (1 number). Actions are stored in joint space and in end-effector space, so users can choose a format.
Evidence in the paper: §III-A · Figure 2 · Appendix B

Check your understanding: What does using the same robot everywhere cost?

The dataset is for one embodiment. Policies trained on it are not shown to transfer to other robots, and the paper does not test that.

2. A collection protocol that forces variety

Intuition

Left alone, collectors repeat what is easy; the interface makes them do something else.

The problem it solves

Common mistakes in crowd-collected data are cameras that cannot see the robot, an operator in the picture, and the same easy task repeated a hundred times in the same spot.

How it works

A session starts by moving the station to a new scene, which collectors are encouraged to choose with several possible tasks, many objects and clutter. They place the cameras so that they capture a wide range of behaviours, calibrate them with a checkerboard, and enter every feasible task for the scene in the interface, picking from a list or typing free text. During collection the interface draws a task at random for each new demonstration, and periodically asks for a “scene augmentation”: nudge the mobile base, move and re-calibrate the cameras, change the room lighting, add or remove objects. A collector typically records up to 100 trajectories, about 20 minutes, in one scene before moving on.

Example

Illustration, not from the paper The interface says “Open the drawer” though the collector was about to pick up a cup again; a few demonstrations later it says “Turn on the lamp” and suggests moving the left camera.

Why the authors chose it

The authors credit this protocol for the diversity measured in Part C: tasks, objects, viewpoints and locations in the workspace.

Technical details

Our calculation By our arithmetic the 350 hours are 16.6 seconds per trajectory, about 249 steps at 15 Hz; a 20-minute session therefore yields about 72 trajectories, which agrees with “up to 100”. The 76 thousand trajectories come to 135 per scene on average, and the paper counts 2,080 scene identifiers that were merged into 564 unique scenes.

Run the example: how big is the lead

Our calculation

What each part does

Type in Table I and divide DROID’s scene and verb counts by each dataset’s.
Derive trajectory length, trajectories per scene and per collector.
Add up the scene types read from the bar chart of Figure 4.

# Table I and the text: how big is DROID's lead, and in what? Everything below is arithmetic on printed numbers.
table1 = {  # dataset: (trajectories, verbs, scenes) as printed in Table I
 "MIME": (8300, 20, 1), "RoboTurk": (2100, 2, 1), "RoboNet": (162000, None, 10), "MT-Opt": (800000, 2, 1), "BridgeData": (7200, 4, 12), "BC-Z": (26000, 3, 1), "RT-1": (130000, 2, 2),
 "RH20T": (13000, 33, 7), "RoboSet": (98500, 9, 11), "BridgeData V2": (60100, 82, 24), "DobbE (tool-based, no robot)": (5600, 6, 216), "Open X-Embodiment": (1400000, 217, 311), "DROID": (76000, 86, 564)}
d = table1["DROID"]
print("scene counts: DROID 564 against each dataset")
for k, (t, v, s) in table1.items():
    if k != "DROID": print(f"  {k:30s} {s:4d} scenes  DROID has {d[2] / s:6.1f} times as many")
print("verb counts: DROID 86 against the datasets with the most verbs")
for k in ("Open X-Embodiment", "BridgeData V2", "RH20T", "RoboSet"):
    print(f"  {k:30s} {table1[k][1]:4d} verbs   DROID has {d[1] / table1[k][1]:.2f} times as many")
print()
print("what the headline numbers imply")
traj, hours = 76000, 350
sec = hours * 3600 / traj
print(f"  {hours} hours over {traj:,} trajectories = {sec:.1f} s per trajectory = {sec * 15:.0f} steps at 15 Hz")
print(f"  trajectories per scene: {traj / 564:.0f} on average; per verb: {traj / 86:.0f}")
print(f"  successful {traj:,} + about 16,000 marked unsuccessful = {traj + 16000:,} recorded, {100 * traj / (traj + 16000):.1f}% successful")
print(f"  scene IDs 2,080 -> 564 unique scenes after merging duplicates ({100 * 564 / 2080:.0f}%); {traj / 2080:.1f} trajectories per scene ID")
print(f"  collection sessions are 'up to 100 trajectories or about 20 minutes': 20 min / {sec:.1f} s = {20 * 60 / sec:.0f} trajectories")
print(f"  per collector (50): {traj / 50:,.0f} trajectories; per robot (18): {traj / 18:,.0f}; 12 months: {traj / 12:,.0f} a month")
print(f"  camera viewpoints: 1,417 over 564 scenes = {1417 / 564:.1f} per scene")
print()
print("scene types, read by eye from the bar chart of Figure 4 (approximate, not printed in the paper)")
bars = [("Industrial office", 205), ("Home kitchen", 135), ("Industrial kitchen", 70), ("Office", 45), ("Living room", 40), ("Hallway / closet", 19), ("Bedroom", 18), ("Dining room", 16), ("Bathroom", 13), ("Laundry room", 3), ("Unknown", 1)]
tot = sum(b for _, b in bars)
print(f"  eleven bars add up to about {tot} (564 scenes are reported)")
home = sum(b for n, b in bars if n in ("Home kitchen", "Living room", "Bedroom", "Dining room", "Bathroom", "Laundry room"))
print(f"  home-type rooms about {home} ({100 * home / tot:.0f}%), industrial or office-type about {tot - home - 1} ({100 * (tot - home - 1) / tot:.0f}%), the largest single type is industrial office ({100 * 205 / tot:.0f}%)")

Output:

scene counts: DROID 564 against each dataset
  MIME                              1 scenes  DROID has  564.0 times as many
  RoboTurk                          1 scenes  DROID has  564.0 times as many
  RoboNet                          10 scenes  DROID has   56.4 times as many
  MT-Opt                            1 scenes  DROID has  564.0 times as many
  BridgeData                       12 scenes  DROID has   47.0 times as many
  BC-Z                              1 scenes  DROID has  564.0 times as many
  RT-1                              2 scenes  DROID has  282.0 times as many
  RH20T                             7 scenes  DROID has   80.6 times as many
  RoboSet                          11 scenes  DROID has   51.3 times as many
  BridgeData V2                    24 scenes  DROID has   23.5 times as many
  DobbE (tool-based, no robot)    216 scenes  DROID has    2.6 times as many
  Open X-Embodiment               311 scenes  DROID has    1.8 times as many
verb counts: DROID 86 against the datasets with the most verbs
  Open X-Embodiment               217 verbs   DROID has 0.40 times as many
  BridgeData V2                    82 verbs   DROID has 1.05 times as many
  RH20T                            33 verbs   DROID has 2.61 times as many
  RoboSet                           9 verbs   DROID has 9.56 times as many

what the headline numbers imply
  350 hours over 76,000 trajectories = 16.6 s per trajectory = 249 steps at 15 Hz
  trajectories per scene: 135 on average; per verb: 884
  successful 76,000 + about 16,000 marked unsuccessful = 92,000 recorded, 82.6% successful
  scene IDs 2,080 -> 564 unique scenes after merging duplicates (27%); 36.5 trajectories per scene ID
  collection sessions are 'up to 100 trajectories or about 20 minutes': 20 min / 16.6 s = 72 trajectories
  per collector (50): 1,520 trajectories; per robot (18): 4,222; 12 months: 6,333 a month
  camera viewpoints: 1,417 over 564 scenes = 2.5 per scene

scene types, read by eye from the bar chart of Figure 4 (approximate, not printed in the paper)
  eleven bars add up to about 565 (564 scenes are reported)
  home-type rooms about 225 (40%), industrial or office-type about 339 (60%), the largest single type is industrial office (36%)

Arithmetic on Table I and the text; the scene-type counts are read by eye from a chart.

Evidence in the paper

§III-B · Figure 11 · Appendix D

Check your understanding: Why is the task drawn at random and not chosen by the collector?

Collectors prefer easy tasks and close objects. A random draw from the list of possible tasks removes that bias and gives each feasible task coverage.

3. What is recorded, and the labels added afterwards

Intuition: An episode is more useful with a sentence, a success flag and the camera geometry attached.
The problem it solves: Raw sensor logs do not say what the demonstrator was trying to do, whether it worked, or where the cameras were.
How it works: The collector marks each episode as a success or not during recording. Afterwards, crowd workers on a labelling platform write up to three independent natural-language instructions per episode, to diversify the wording. Each episode also stores the building name and the collector’s identifier, and a scene type assigned by a vision-language model, GPT-4V, that was prompted with a random frame. About 16 thousand episodes were marked unsuccessful; they are released but not counted in the 76 thousand.
Example: Illustration, not from the paper One episode may carry “put the apple in the pot”, “place the fruit into the pan” and “move the apple into the pot and close it”.
Why the authors chose it: Language-conditioned policies need instructions, and users can filter by success, building or scene type.
Technical details: Extrinsic calibration (where each camera is relative to the robot) comes from the checkerboard step at the start of each session, which can be wrong; the next idea describes how the authors improve it after the fact.
Evidence in the paper: §III-B · Appendix B · Appendix C

Check your understanding: Are the unsuccessful episodes part of the “76 thousand”?

No. The size counts only successful episodes. The roughly 16 thousand others are in the release but excluded from the count, and were excluded from the training mixture in the experiments.

4. Fixing camera calibration after the fact

Intuition

A checkerboard in a hurry is often a little wrong; a model can check it.

The problem it solves

Calibration errors make 3D reasoning, such as learning from the geometry of the scene, unreliable.

How it works

Appendix G describes three automatic checks. First, the robot’s known joint angles are used to render its silhouette and project key points into the image, and the result is compared with a mask found by a segmentation model (Segment Anything); good alignments are kept. Second, a keypoint network (CtRNet-X) is run on all videos and scenes with a small reprojection error are kept. Third, a 3D reconstruction network (DUSt3R), modified to use the recorded depth, estimates the pose of one external camera relative to the other. The release offers three sets of calibration matrices.

Example

Illustration, not from the paper A scene where the checkerboard was misplaced is detected because the rendered robot outline does not overlap the segmented robot in the image; its calibration is dropped from the “good” set.

Why the authors chose it

It lets users trust the geometry for a known subset, around 36 thousand trajectories with one camera calibrated to the robot base and about 24 thousand with both.

Technical details

Our calculation The two filters pass about 30 thousand and 12 thousand, and the union is “around 36 thousand”, so they overlap by about 6 thousand. The paper calls these “unique scenes”, a number far larger than the 564 unique scenes; the counts fit trajectories better (Part F). Running the two filters took about 1,152 card-hours on A100 cards.

Run the example: calibration counts

Our calculation

What each part does

Type in the printed counts of the two filters and their union.
Estimate the overlap and the compute used.
Compare “scenes” in this appendix with the 564 scenes of the main text.

# Appendix G: post-hoc camera calibration. Counts as printed; the rest is arithmetic.
base_a, base_b, union = 30000, 12000, 36000     # well-calibrated by the keypoint-mask check (G-A), by CtRNet-X (G-B), and together
print(f"G-A about {base_a:,}, G-B about {base_b:,}, union 'around 36k': overlap about {base_a + base_b - union:,} ({100 * (base_a + base_b - union) / base_b:.0f}% of the G-B set)")
print(f"G-A took about 1 day on 8 A100 cards = {1 * 24 * 8} card-hours; G-B about 5 days = {5 * 24 * 8} card-hours; total {6 * 24 * 8:,} card-hours")
print(f"per scene (36,000): {(6 * 24 * 8) * 3600 / 36000:.0f} card-seconds")
print()
print("what 'scene' means in this appendix")
print(f"  36,000 'unique scenes' against 564 unique scenes in the paper: {36000 / 564:.0f} times more; 76,000 trajectories / 36,000 = {76000 / 36000:.1f}")
print(f"  24,000 with both cameras calibrated = {100 * 24 / 36:.0f}% of 36,000. The counts lie between 564 and 76,000, so our reading is that they count trajectories (our interpretation; the paper does not define them)")
print(f"  as trajectories: {100 * 36 / 76:.0f}% of the 76,000 have one calibrated camera and {100 * 24 / 76:.0f}% have both")
print()
print("thresholds used for the curated set: intersection over union >= 0.6 (0.7 in G-A), reprojection error <= 20, top 30% of matches")

Output:

G-A about 30,000, G-B about 12,000, union 'around 36k': overlap about 6,000 (50% of the G-B set)
G-A took about 1 day on 8 A100 cards = 192 card-hours; G-B about 5 days = 960 card-hours; total 1,152 card-hours
per scene (36,000): 115 card-seconds

what 'scene' means in this appendix
  36,000 'unique scenes' against 564 unique scenes in the paper: 64 times more; 76,000 trajectories / 36,000 = 2.1
  24,000 with both cameras calibrated = 67% of 36,000. The counts lie between 564 and 76,000, so our reading is that they count trajectories (our interpretation; the paper does not define them)
  as trajectories: 47% of the 76,000 have one calibrated camera and 32% have both

thresholds used for the curated set: intersection over union >= 0.6 (0.7 in G-A), reprojection error <= 20, top 30% of matches

Arithmetic on the printed counts; the “trajectory” reading of the counts is ours.

Evidence in the paper

§III-B · Appendix G · Figures 13 to 17

Check your understanding: Is the extra calibration used in the paper’s policy experiments?

No. The policies use the two external images, proprioception and a language embedding; the calibration is released for other uses such as 3D perception.

5. The policy used to test the data

Intuition

Use a standard, well-tested learner, so differences come from the data.

The problem it solves

To show that a dataset helps, the learning algorithm must be fixed and unremarkable.

How it works

The paper uses a diffusion policy, in the Robomimic implementation. Each of the two external images is reduced to 128 × 128 pixels and encoded by an ImageNet-pretrained ResNet-50; the embeddings are joined with a frozen DistilBERT embedding of the instruction, the gripper position and the gripper state, passed through a multilayer perceptron (MLP) and then to a U-Net diffusion head. The policy outputs 16-step sequences of absolute end-effector actions, of which 8 are executed before it plans again, using the DDIM sampler.

Example

Illustration, not from the paper At 15 Hz the robot executes 8 actions, a bit over half a second of motion, then looks again.

Why the authors chose it

A diffusion policy was the strong standard for imitation at the time, and the authors want the dataset, not a new learner, to be the contribution.

Technical details

Table II and Appendix F, as printed.
Setting	Value
Batch size, steps	128; 25,000 (50,000 for the Cook Lentils OOD runs)
Optimiser, learning rate	Adam, 1e-4, linear schedule
Observation, prediction, action horizon	2, 16, 8
Image size, crop	128 × 128, 116 × 116 random crop with colour jitter
Observation MLP	[1024, 512, 512]
U-Net hidden sizes	[256, 512, 1024]

Evidence in the paper

§V-A · Appendix F-A · Table II

Check your understanding: The policies receive a language instruction. Do the experiments test language following?

Not directly: a separate policy is trained for each task with one instruction, so the tasks do not need language to be told apart. The instruction input is present but its effect is not tested.

Part C

Understand the data and the training

Two things need care: what “diverse” means and how it is measured, and how the dataset is mixed into training. The paper offers a measurement for the first and a simple recipe for the second.

What “diverse” means: five axes and how they are measured

Intuition

Different kinds of variety help with different kinds of new situations.

The problem it solves

Datasets are called “diverse” without saying in what, so claims cannot be compared.

How it works

The authors pick five axes. Task diversity is the number of distinct verbs in the instructions, found with a semantic parser and de-duplicated with GPT-4. Object diversity is the objects the robot manipulates, from the same parse. Scene diversity is the number of unique workspaces, counted by the authors’ definition, with a scene type per scene from GPT-4V. Viewpoint diversity is the number of distinct camera poses, 1,417. Interaction-location diversity is the 3D position of the gripper where it first closes, as a proxy for where in the workspace things happen. They compare DROID with RT-1, RH20T and Bridge V2.

Example

Illustration, not from the paper A dataset of 130 thousand episodes of the same two verbs in two scenes is large but not diverse on the first and third axes; DROID has 86 verbs over 564 scenes.

Why the authors chose it

It makes the diversity claim checkable and ties it to the kinds of generalisation each axis could support.

Technical details

Our calculation DROID has 23.5 times as many scenes as Bridge V2 and 80.6 times as many as RH20T, but only 1.8 times as many as the 311 scenes the paper lists for Open X-Embodiment, and fewer verbs (86 against 217). “An order of magnitude” holds against the single-site datasets, not against the pooled collection.

Our interpretation The scene types are shown in a bar chart without printed counts; read by eye, about 36% of the scenes are industrial offices and about 40% are home-type rooms.

Evidence in the paper

§IV · Figures 3 to 6 · Table I

Check your understanding: Why does the paper plot verbs on a logarithmic scale?

Because diversity is about covering many tasks, not about how many episodes each has: 0 against 10 examples matters more than 1,000 against 2,000.

Definitions that decide the numbers: task, scene, success

Intuition: A count is only as meaningful as the definition behind it.
The problem it solves: Earlier papers define “task” and “scene” differently, so the counts in a comparison table are not comparable unless the definitions are stated.
How it works: In this paper a task is a verb (so “86 tasks” means 86 distinct verbs), not a verb with an object. A scene is counted as new only when the robot’s workspace changes substantially, such as a different corner of a kitchen; moving objects or the table cloth does not count. During collection a scene identifier is created each time the collector indicates that the robot or cameras moved, giving 2,080 identifiers, which the authors then merged by hand into 564 by grouping by robot, lab and building and removing repeats. They describe this count as conservative.
Example: Illustration, not from the paper The robot in front of the sink and in front of the stove in the same kitchen are two scenes; the same sink with a rearranged counter is one.
Why the authors chose it: It is the reason the 564 can be compared with other datasets’ scene counts, which the authors say they checked manually or with the dataset authors.
Technical details: The merge of 2,080 identifiers into 564 scenes was done by hand and is not reproducible from the paper; the authors note that two different robots at the same place would be counted twice.
Evidence in the paper: §II (footnote 1) · Appendix D · Table I

Check your understanding: If one verb had 3 different objects, how many “tasks” would the paper count?

One. Tasks are counted as verbs, so “pick up the cup” and “pick up the shoe” are the same task.

Using it: 50/50 co-training

Intuition

Show the small task dataset and the large background dataset in equal parts.

The problem it solves

Fifty demonstrations are too few to learn a task robustly, and a large dataset from other situations cannot simply replace them.

How it works

For each task the authors train three policies that differ only in how every training batch is built. “No co-training”: all samples from the task demonstrations. “DROID”: half the batch from the task demonstrations and half from DROID. “OXE”: half and half with a curated part of the Open X-Embodiment (OXE) collection, the one used by Octo, with Language Table removed, about 400 thousand trajectories. Only successful DROID episodes with language annotations available at the time were used, the first 40 thousand.

Example

Illustration, not from the paper A batch of 128 has 64 samples from the 50 waffle-maker demonstrations and 64 from DROID.

Why the authors chose it

The authors report that this simple mix worked well in practice, and it keeps all methods identical apart from the data.

Technical details

Our calculation Assuming an in-domain demonstration is as long as the DROID average (249 steps; the paper does not say), 25,000 steps of 64 task samples show each task time step about 43 to 129 times, while each DROID step is seen about 0.16 times, so the task data are repeated hundreds of times more often. The OXE set has 10 times as many trajectories as the DROID subset it is compared with.

Run the example: co-training arithmetic

Our calculation

What each part does

Work out the samples drawn from each source in 25,000 steps.
Estimate passes over each source, with the length assumption stated.
Compare the sizes of the three data sources.

# 50/50 co-training: how often is each kind of data seen? Section V-A and Appendix F (Table II). Assumptions are marked.
batch, steps = 128, 25000
in_domain_per_batch = batch // 2
seen = in_domain_per_batch * steps
print(f"batch {batch}, {steps:,} steps -> {batch * steps:,} samples; half ({seen:,}) from the in-domain demonstrations")
sec, hz = 350 * 3600 / 76000, 15
steps_per_traj = sec * hz
print(f"assumption (not in the paper): an in-domain demonstration is as long as the DROID average, {steps_per_traj:.0f} steps")
print()
print("passes over the data, counting one sample per time step (an approximation: the paper does not say how samples are drawn)")
for demos in (50, 60, 70, 150):
    total = demos * steps_per_traj
    print(f"  {demos:3d} demonstrations = {total:8,.0f} steps -> each step is drawn about {seen / total:6.0f} times")
droid40 = 40000 * steps_per_traj
print(f"  DROID, first 40,000 trajectories = {droid40:,.0f} steps -> {seen / droid40:.2f} passes (it supplies the other half of every batch)")
print(f"  so a DROID step is drawn about {seen / droid40:.2f} times and an in-domain step about {seen / (150 * steps_per_traj):.0f} to {seen / (50 * steps_per_traj):.0f} times: roughly {(seen / (150 * steps_per_traj)) / (seen / droid40):.0f} to {(seen / (50 * steps_per_traj)) / (seen / droid40):.0f} times as often")
print(f"  Open X-Embodiment baseline: about 400,000 trajectories, {400000 / 40000:.0f} times as many as the DROID set it is compared with")
print()
print("data compared in Figure 8 (trajectories): in-domain 50 to 150 demonstrations | DROID 40,000 (of 76,000 recorded) | Open X-Embodiment 400,000 (Language Table removed)")
print(f"ratio of DROID to in-domain data: {40000 / 150:.0f} to {40000 / 50:.0f} times; of Open X-Embodiment to DROID: {400000 / 40000:.0f} times")

Output:

batch 128, 25,000 steps -> 3,200,000 samples; half (1,600,000) from the in-domain demonstrations
assumption (not in the paper): an in-domain demonstration is as long as the DROID average, 249 steps

passes over the data, counting one sample per time step (an approximation: the paper does not say how samples are drawn)
   50 demonstrations =   12,434 steps -> each step is drawn about    129 times
   60 demonstrations =   14,921 steps -> each step is drawn about    107 times
   70 demonstrations =   17,408 steps -> each step is drawn about     92 times
  150 demonstrations =   37,303 steps -> each step is drawn about     43 times
  DROID, first 40,000 trajectories = 9,947,368 steps -> 0.16 passes (it supplies the other half of every batch)
  so a DROID step is drawn about 0.16 times and an in-domain step about 43 to 129 times: roughly 267 to 800 times as often
  Open X-Embodiment baseline: about 400,000 trajectories, 10 times as many as the DROID set it is compared with

data compared in Figure 8 (trajectories): in-domain 50 to 150 demonstrations | DROID 40,000 (of 76,000 recorded) | Open X-Embodiment 400,000 (Language Table removed)
ratio of DROID to in-domain data: 267 to 800 times; of Open X-Embodiment to DROID: 10 times

Arithmetic on Table II and Appendix F; the in-domain episode length is our assumption.

Evidence in the paper

§V-A · §V-B · Appendix F-B

Check your understanding: The main text says “the complete DROID dataset” is used. What does the appendix say?

That the first 40 thousand successful trajectories with language annotations available at the time were used, and a 7,362-trajectory subset of those for the scene-diversity test.

Part D

Understand the experiments

Two questions: does co-training with DROID improve a policy compared with the task data alone and compared with Open X-Embodiment, and how much of that is due to scene diversity? This part explains the tests and what the figures say. Our re-calculations are in the folded examples.

The set-up: six tasks, in-distribution and out-of-distribution

How it works

Six tasks in four places. Short: Close Waffle Maker (70 demonstrations) and Place Chips on Plate (50), both in a lab. Medium: Put Apple in Pot (60) and Toasting (150, a multi-task policy shared with the waffle maker), in the lab. Long: Clean up Desk (50, in an office: open a drawer, put an eraser in, close it) and Cook Lentils (50, in a kitchen: take off the lid, pour lentils, turn on the stove). Each is run in-distribution, with randomised start positions, and in an out-of-distribution (OOD) variant: distractor objects, a different chips bag, new objects to toast, or a camera shift. Chips has two OOD variants, so there are 7 OOD settings. Every method gets 10 rollouts per setting; initial positions are matched across methods.

Technical details

Our calculation That is 60 rollouts per method in distribution and 70 out of distribution. With 10 rollouts a single bar has a standard error of 16 points; with 60 or 70 rollouts a gap must be about 19 to 20 points before an exact test calls it real, and with 30 about 30.

Run the example: how much a success rate can move by chance

Our calculation

What each part does

Take the trial counts used in the paper: 10, 30, 60 and 70.
Compute the standard error of a rate near 50%.
Find the smallest gap an exact two-sided test treats as different, and test the headline gaps.

import math
# How much can a success rate move by chance? Trial counts used in the DROID paper: 10 rollouts per task setting.
def se(p, n): return 100 * math.sqrt(p * (1 - p) / n)
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
print("rollouts | where                                          | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)")
for n, where in ((10, "one task setting, one method"), (30, "Figure 10: three settings"), (60, "Figure 8, in distribution: six tasks"), (70, "Figure 8, out of distribution: seven settings")):
    base = n // 2
    gap = next((100 * (k - base) / n for k in range(base, n + 1) if fisher(k, n, base, n) < 0.05), None)
    print(f"{n:8d} | {where:46s} | {se(0.5, n):9.1f} pts | {gap:.0f} points")
print()
print("headline gaps as exact tests on pooled rollouts:")
for name, a, b, n in (("in distribution: DROID 45/60 against no co-training 32/60", 45, 32, 60), ("in distribution: DROID 45/60 against OXE 26/60", 45, 26, 60), ("out of distribution: DROID 37/70 against OXE 25/70", 37, 25, 70),
                      ("out of distribution: DROID 37/70 against no co-training 12/70", 37, 12, 70), ("scene diversity: diverse 18/30 against 20 scenes 12/30", 18, 12, 30)):
    print(f"  {name:62s} {100 * a / n:5.1f}% vs {100 * b / n:5.1f}%: p = {fisher(a, n, b, n):.4f}")
print()
print("one task setting, 10 rollouts: 95% interval of the true rate (Wilson) for the counts in Figure 8")
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d; h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return 100 * (c - h), 100 * (c + h)
for k in (1, 3, 6, 9):
    lo, hi = wilson(k, 10); print(f"  {k}/10 -> {lo:3.0f}% to {hi:3.0f}%")

Output:

rollouts | where                                          | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)
      10 | one task setting, one method                   |      15.8 pts | 50 points
      30 | Figure 10: three settings                      |       9.1 pts | 30 points
      60 | Figure 8, in distribution: six tasks           |       6.5 pts | 20 points
      70 | Figure 8, out of distribution: seven settings  |       6.0 pts | 19 points

headline gaps as exact tests on pooled rollouts:
  in distribution: DROID 45/60 against no co-training 32/60       75.0% vs  53.3%: p = 0.0218
  in distribution: DROID 45/60 against OXE 26/60                  75.0% vs  43.3%: p = 0.0007
  out of distribution: DROID 37/70 against OXE 25/70              52.9% vs  35.7%: p = 0.0609
  out of distribution: DROID 37/70 against no co-training 12/70   52.9% vs  17.1%: p = 0.0000
  scene diversity: diverse 18/30 against 20 scenes 12/30          60.0% vs  40.0%: p = 0.1964

one task setting, 10 rollouts: 95% interval of the true rate (Wilson) for the counts in Figure 8
  1/10 ->   2% to  40%
  3/10 ->  11% to  60%
  6/10 ->  31% to  83%
  9/10 ->  60% to  98%

Our calculation, assuming independent trials; each experiment is a single run.

Evidence in the paper

§V-A · §V-B · Appendix E

Check your understanding: Why are policies trained separately for each task?

The aim is to see whether extra data improves a task-specific policy; it does not test a single language-conditioned policy for many tasks, apart from the one policy shared by Toasting and Close Waffle Maker.

Does DROID improve performance and robustness? (Figure 8)

How it works

In distribution the averages over six tasks are 0.75 for DROID co-training, 0.53 without co-training and 0.43 with Open X-Embodiment. Out of distribution, over seven settings, they are 0.53, 0.17 and 0.36. The authors say DROID beats the next best method by 22 points in distribution and 17 out of distribution, and note that DROID is the only method that finishes all three stages of Cook Lentils consistently.

Technical details

Figure 8, success rate over 10 rollouts per entry, as printed on the bars.
Setting	No co-training	OXE	DROID
Close Waffle Maker	0.8	0.7	0.9
Put Chips in Plate	0.6	0.4	0.7
Put Apple in Pot	0.3	0.1	0.4
Toasting	0.7	0.7	0.8
Clean up Desk	0.7	0.6	1.0
Cook Lentils	0.1	0.1	0.7
In-distribution average	0.53	0.43	0.75
Waffle Maker (distractors)	0.4	0.5	0.7
Chips (distractors)	0.1	0.4	0.4
Chips (novel chips)	0.2	0.6	0.3
Apple in Pot (distractors)	0.0	0.1	0.2
Toasting (novel objects)	0.2	0.3	0.7
Clean up Desk (distractors)	0.0	0.6	0.8
Cook Lentils (distractors + camera shift)	0.3	0.0	0.6
OOD average	0.17	0.36	0.53

Our calculation The averages and both gaps (21.7 and 17.1 points) reproduce, and the figure’s error bars match the standard error across tasks (our calculation gives 0.085 for DROID in distribution). DROID is better than both baselines in 6 of 6 in-distribution tasks (sign test p of 0.031). Out of distribution it beats no co-training in 7 of 7 settings, but beats Open X-Embodiment in 5 settings, ties in 1 and loses in 1 (the novel chips bag, 0.3 against 0.6); the sign test gives p of 0.219 and the pooled counts 37/70 against 25/70 give an exact p of 0.0609.

Run the example: recomputing the averages and testing the gaps

Our calculation

What each part does

Type in the values printed on the bars and recompute the averages and standard errors.
Count the settings where DROID wins, ties or loses and run a sign test.
Pool the rollouts and run an exact test.

import math
# Figure 8: the bar values printed on the chart (success rate), 10 rollouts per task setting and method.
methods = ["No co-training", "OXE", "DROID"]
ind = {"Close Waffle Maker": (0.8, 0.7, 0.9), "Put Chips in Plate": (0.6, 0.4, 0.7), "Put Apple in Pot": (0.3, 0.1, 0.4), "Toasting": (0.7, 0.7, 0.8), "Clean up Desk": (0.7, 0.6, 1.0), "Cook Lentils": (0.1, 0.1, 0.7)}
ood = {"Close Waffle Maker (distractors)": (0.4, 0.5, 0.7), "Put Chips in Plate (distractors)": (0.1, 0.4, 0.4), "Put Chips in Plate (novel chips)": (0.2, 0.6, 0.3), "Put Apple in Pot (distractors)": (0.0, 0.1, 0.2),
       "Toasting (novel objects)": (0.2, 0.3, 0.7), "Clean up Desk (distractors)": (0.0, 0.6, 0.8), "Cook Lentils (distractors + camera shift)": (0.3, 0.0, 0.6)}
printed = {"in": (0.53, 0.43, 0.75), "ood": (0.17, 0.36, 0.53)}
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
def sign_p(pos, neg):  # exact two-sided sign test, ties dropped
    n = pos + neg; k = max(pos, neg)
    return min(1.0, 2 * sum(math.comb(n, i) for i in range(k, n + 1)) / 2 ** n)
for name, tab, key in (("in distribution (6 tasks, 60 rollouts per method)", ind, "in"), ("out of distribution (7 settings, 70 rollouts per method)", ood, "ood")):
    n = len(tab); print(name)
    means = [sum(v[j] for v in tab.values()) / n for j in range(3)]
    for j, m in enumerate(methods):
        sd = math.sqrt(sum((v[j] - means[j]) ** 2 for v in tab.values()) / (n - 1))
        print(f"  {m:15s} mean {means[j]:.3f} (printed {printed[key][j]})  standard error across tasks {sd / math.sqrt(n):.3f}")
    print(f"  DROID minus the next best: {100 * (means[2] - max(means[:2])):.1f} points; minus no co-training {100 * (means[2] - means[0]):.1f}")
    for j, m in enumerate(methods[:2]):
        pos = sum(v[2] > v[j] for v in tab.values()); neg = sum(v[2] < v[j] for v in tab.values()); tie = n - pos - neg
        print(f"  DROID against {m:15s}: better in {pos}, equal in {tie}, worse in {neg} settings; sign test p = {sign_p(pos, neg):.3f}")
        a = round(means[2] * 10 * n); b = round(means[j] * 10 * n)
        print(f"     pooled rollouts: {a}/{10 * n} against {b}/{10 * n}, exact p = {fisher(a, 10 * n, b, 10 * n):.4f}")
    print()
print("the paper says 'about 20% on average' in the introduction; the Figure 8 caption says 22% in distribution and 17% out of distribution:")
print(f"  mean of the two gaps over the next best method: {(22 + 17) / 2:.1f}")
w = [(k, v) for k, v in ood.items() if v[2] < max(v[:2])]
print("out-of-distribution settings where DROID is not the best method:", [(k, v) for k, v in w])

Output:

in distribution (6 tasks, 60 rollouts per method)
  No co-training  mean 0.533 (printed 0.53)  standard error across tasks 0.112
  OXE             mean 0.433 (printed 0.43)  standard error across tasks 0.115
  DROID           mean 0.750 (printed 0.75)  standard error across tasks 0.085
  DROID minus the next best: 21.7 points; minus no co-training 21.7
  DROID against No co-training : better in 6, equal in 0, worse in 0 settings; sign test p = 0.031
     pooled rollouts: 45/60 against 32/60, exact p = 0.0218
  DROID against OXE            : better in 6, equal in 0, worse in 0 settings; sign test p = 0.031
     pooled rollouts: 45/60 against 26/60, exact p = 0.0007

out of distribution (7 settings, 70 rollouts per method)
  No co-training  mean 0.171 (printed 0.17)  standard error across tasks 0.057
  OXE             mean 0.357 (printed 0.36)  standard error across tasks 0.090
  DROID           mean 0.529 (printed 0.53)  standard error across tasks 0.087
  DROID minus the next best: 17.1 points; minus no co-training 35.7
  DROID against No co-training : better in 7, equal in 0, worse in 0 settings; sign test p = 0.016
     pooled rollouts: 37/70 against 12/70, exact p = 0.0000
  DROID against OXE            : better in 5, equal in 1, worse in 1 settings; sign test p = 0.219
     pooled rollouts: 37/70 against 25/70, exact p = 0.0609

the paper says 'about 20% on average' in the introduction; the Figure 8 caption says 22% in distribution and 17% out of distribution:
  mean of the two gaps over the next best method: 19.5
out-of-distribution settings where DROID is not the best method: [('Put Chips in Plate (novel chips)', (0.2, 0.6, 0.3))]

Arithmetic on the printed values, each a count out of 10.

Evidence in the paper

§V-B · Figure 8 · Appendix E

Check your understanding: Which of the two headline gaps, 22 or 17 points, is better supported?

The 22 points in distribution: DROID wins all six tasks over both baselines. The 17 points out of distribution is mostly the comparison with Open X-Embodiment, where one setting reverses and the pooled test is just above the usual 0.05 line.

How important is scene diversity? (Figure 10)

How it works

On the three hardest out-of-distribution settings (novel toasting objects, desk distractors, lentils with distractors and a camera shift) two subsets of DROID of equal size, 7,362 trajectories, are compared. One takes the 20 scenes with the most demonstrations; the other is a uniform random sample. The random one averages 0.60 and the 20-scene one 0.40. The full dataset scores 0.70 on the same three settings. The authors conclude that the strength of DROID lies in its size and especially its diversity.

Technical details

Our calculation 18 against 12 successes of 30 has an exact p of 0.196, so the difference is within chance; the three per-task differences are all positive (0.3, 0.1, 0.2). The 20-scene subset (12/30) is level with the whole of Open X-Embodiment (9/30, p of 0.589) on these settings, even though it is about 54 times smaller. The paper does not report how many scenes the random subset contains, or the number of distinct tasks or objects in either subset.

Run the example: scene diversity against 20 scenes

Our calculation

What each part does

Type in the bar values and combine them with Figure 8 for the same settings.
Test the diverse subset against the 20-scene subset and the 20-scene subset against Open X-Embodiment.
Work out what fraction of the dataset each subset is.

import math
# Figure 10: co-training with 7,362 DROID trajectories from the 20 scenes with the most data, or 7,362 random ones. Out-of-distribution, 10 rollouts per task.
tasks = ["Toasting (novel objects)", "Clean up Desk (distractors)", "Cook Lentils (distractors + camera shift)"]
twenty, diverse = (0.3, 0.7, 0.2), (0.6, 0.8, 0.4)
full = (0.7, 0.8, 0.6)             # co-training with the full set (Figure 8, same three settings)
oxe = (0.3, 0.6, 0.0); none = (0.2, 0.0, 0.3)
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
avg = lambda v: sum(v) / len(v)
print("means over the three out-of-distribution settings (30 rollouts each):")
for n, v in (("no co-training", none), ("full Open X-Embodiment (about 400 thousand trajectories)", oxe), ("DROID, 7k from 20 scenes", twenty), ("DROID, 7k random (diverse scenes)", diverse), ("DROID, first 40k", full)):
    print(f"  {n:58s} {avg(v):.2f}  ({round(avg(v) * 30)}/30)")
a, b = round(avg(diverse) * 30), round(avg(twenty) * 30)
print(f"diverse against 20 scenes: {a}/30 against {b}/30, exact p = {fisher(a, 30, b, 30):.3f}; per task {[round(x - y, 1) for x, y in zip(diverse, twenty)]}")
c = round(avg(twenty) * 30); o = round(avg(oxe) * 30)
print(f"20-scene subset against all of Open X-Embodiment: {c}/30 against {o}/30, exact p = {fisher(c, 30, o, 30):.3f}")
print()
print("how much of DROID is each subset?")
print(f"  7,362 of 40,000 = {100 * 7362 / 40000:.1f}% of the annotated trajectories used in training; of 76,000 = {100 * 7362 / 76000:.1f}%")
print(f"  20 scenes are {100 * 20 / 564:.1f}% of the 564 scenes; 7,362 / 20 = {7362 / 20:.0f} trajectories per scene against {76000 / 564:.0f} on average")
print(f"  trajectories left for the other 544 scenes (if all 76,000 are counted): {76000 - 7362:,}, {(76000 - 7362) / 544:.0f} per scene")
print("  the paper does not say how many scenes the random 7,362 contain")

Output:

means over the three out-of-distribution settings (30 rollouts each):
  no co-training                                             0.17  (5/30)
  full Open X-Embodiment (about 400 thousand trajectories)   0.30  (9/30)
  DROID, 7k from 20 scenes                                   0.40  (12/30)
  DROID, 7k random (diverse scenes)                          0.60  (18/30)
  DROID, first 40k                                           0.70  (21/30)
diverse against 20 scenes: 18/30 against 12/30, exact p = 0.196; per task [0.3, 0.1, 0.2]
20-scene subset against all of Open X-Embodiment: 12/30 against 9/30, exact p = 0.589

how much of DROID is each subset?
  7,362 of 40,000 = 18.4% of the annotated trajectories used in training; of 76,000 = 9.7%
  20 scenes are 3.5% of the 564 scenes; 7,362 / 20 = 368 trajectories per scene against 135 on average
  trajectories left for the other 544 scenes (if all 76,000 are counted): 68,638, 126 per scene
  the paper does not say how many scenes the random 7,362 contain

Arithmetic on the printed bars; one run per bar.

Evidence in the paper

§V-C · Figure 10

Check your understanding: What would make the scene-diversity conclusion firmer?

More rollouts and tasks, a report of how many scenes, objects and tasks each subset contains, and subsets that vary one thing at a time (scenes while holding the number of tasks and objects fixed).

What the qualitative examples add (Figure 9)

How it works: Figure 9 shows one rollout of Cook Lentils per method in an author’s kitchen: the DROID policy completes all four visual stages (remove lid, pick up lentils, pour, turn on stove); the Open X-Embodiment policy completes the first two; the no-co-training policy completes the first. The text says that policies with DROID are smoother and more precise and that baselines tend to fail after one or two steps.
Technical details: The figure is a single example per method and is labelled representative. Smoothness and precision are not measured; the numbers in Figure 8 are success counts only, so the qualitative claim is an observation, not a result.
Evidence in the paper: §V-B · Figure 9

Check your understanding: What kind of evidence is a “representative rollout”?

An illustration chosen by the authors. It shows what a failure or success looks like but not how common it is; the success counts in Figure 8 do that.

What the results show

Adding half-batches of DROID clearly improves a diffusion policy on six real-robot tasks over training on the task data alone, both in distribution and with distractors, new objects or a camera shift.
In distribution, DROID is better than Open X-Embodiment on every task.
A DROID subset of 7 thousand trajectories already matches the whole of Open X-Embodiment on three hard settings.

What they do not show

That the advantage over Open X-Embodiment out of distribution is more than chance (p of 0.219 by sign test, 0.0609 pooled), or that it comes from scene diversity rather than from matching the robot.
That random scenes beat 20 scenes (p of 0.196 with 30 rollouts).
That DROID helps other robots, other policy classes, language following or larger models; none of these was tested.
That the full 76 thousand episodes were used: the experiments use the first 40 thousand annotated ones.

Part E · optional

Engineering deep dive

The comparison table, the recorded fields, the collection interface, the evaluation tasks and the calibration pipeline, for readers who want to use or audit the data. Everything is collapsed; Parts A to D do not depend on it.

Dataset comparison (Table I)

As printed in Table I (RH20T trajectories counted once per multi-view episode, as the paper explains). Public availability is omitted here.
Dataset	Trajectories	Verbs	Scenes	Language	Calibration	Collection
MIME	8.3k	20	1	no	no	human teleoperation
RoboTurk	2.1k	2	1	no	no	human teleoperation
RoboNet	162k	n/a	10	no	no	scripted
MT-Opt	800k	2	1	no	no	scripted and learned
BridgeData	7.2k	4	12	yes	no	human teleoperation
BC-Z	26k	3	1	yes	no	human teleoperation
RT-1	130k	2	2	yes	no	human teleoperation
RH20T	13k	33	7	yes	yes	human teleoperation
RoboSet	98.5k	9	11	yes	no	30% human, 70% scripted
BridgeData V2	60.1k	82	24	yes	no	85% human, 15% scripted
DobbE	5.6k	6	216	yes	n/a	human, tool-based
Open X-Embodiment	1.4M	217	311	partly	no	aggregation of datasets
DROID	76k	86	564	yes	yes	human teleoperation

What is recorded in each episode (Appendix B)

Reported by the paper Three stereo RGB camera streams at 1280 × 720, at 15 Hz.
Reported by the paper Joint positions and velocities (7 numbers), end-effector pose and velocity in the robot base frame (6), gripper position and velocity (1).
Reported by the paper Actions in several spaces (joint and end-effector), from the controller.
Reported by the paper One to three crowd-sourced instructions; extrinsic calibration matrices for both external cameras; building name and collector identifier; a GPT-4V scene type; the success mark.

The collection interface (Figure 11)

Reported by the paper The interface has four screens: a screen to enter the feasible tasks of the scene (from a list of suggestions or free text), an instruction screen that shows a randomly drawn task, a collection screen with the live camera streams (RGB and depth), and a prompt that periodically suggests a scene change. Camera views are chosen by the collector so that a wide range of behaviours is visible and so that the robot and not the operator is in view.

The evaluation tasks (Appendix E)

Reported by the paper Place Chips on Plate (50 demonstrations): success if the chips are on the plate; OOD: a different chips bag (Sun Chips) or two added distractors (an apple and an orange).
Reported by the paper Put Apple in Pot (60): apple in the pot and the lid on; OOD: an extra plate on the table. Toasting (150): object in the toaster oven and the oven closed; OOD: new objects. Close Waffle Maker (70): the waffle maker closed; OOD: several distractors.
Reported by the paper Clean up Desk (50, office): the drawer closed with the eraser in it; OOD: a calculator, three whiteboard markers and a clip on the desk. Putting distractors inside the drawer made every policy fail, so it was not used. Cook Lentils (50, kitchen): all three stages done; OOD: several distractors and a camera shift.

The calibration pipeline (Appendix G)

Reported by the paper Check of the existing calibration: project robot key points with the stored extrinsics, prompt a segmentation model, render the robot with PyTorch3D from its model file, and keep frames whose overlap (intersection over union) is at least 0.7 (0.6 for the final curated set); the model’s confidence must exceed 0.65; the mean over 5 frames is reported. About 30 thousand scenes passed, in about a day on 8 A100 cards.
Reported by the paper Automatic camera-to-robot calibration: CtRNet-X, a keypoint network guided by CLIP, with outlier rejection by median absolute deviation and a mean reprojection error of at most 20; about 12 thousand scenes, in about 5 days on 8 cards.
Reported by the paper Camera-to-camera calibration: DUSt3R with fixed depth and intrinsics for metric scale; the top 30% of scenes by number of matched points are kept. The curated release has about 24 thousand scenes with both cameras calibrated to the base.

Limitations stated by the authors

The discussion lists open questions rather than limitations: how to combine DROID with other datasets, how to train policies for new scenes with no task data, whether the data give better visual representations, and when slices of the data are better than the whole.
For the calibration, the authors note false positives, that one tool was trained only on the Franka arm, and that the pose model fails in clutter or when views barely overlap.

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: where its counts disagree, where a comparison differs in more than the thing being credited, and where small sample sizes limit the conclusions. DROID is a valuable, open and well-documented dataset, and a lot of the care shows in its appendix; the findings concern the strength of the evidence for what it improves. Each one links back to the concept it affects, and the calculations are reproducible in the examples.

Numbers that do not agree

F1. Four different lists of scene types Numerical inconsistency

What the paper says. The text defines 10 scene types for the scene analysis.

What we found. The text says 10 scene types; the bar chart of Figure 4 has 11 labels (industrial office, home kitchen, industrial kitchen, office, living room, hallway or closet, bedroom, dining room, bathroom, laundry room, unknown); the caption of Figure 12 says 9 types and its labels include Laboratory and Closet, which are not in the other lists; and the prompt in Appendix C lists 12 categories, including “Other” and “Unknown”. Read by eye from the chart, the 11 bars add up to about 565, which fits the 564 scenes.

Why it matters. A small inconsistency, but it means the scene-type breakdown cannot be reproduced from the paper, and the labelling was done by a vision model whose agreement with humans is described only as “high quality” after a review. Concept: The five axes · example
F2. Institutions, labs and robots: 13, 18 and 18 Numerical inconsistency

What the paper says. DROID was collected by 50 data collectors at 13 institutions, with 18 robots (Section III), by 18 research labs (Introduction), and the platform “exists in 18 labs” (Discussion).

What we found. The figures are compatible if several labs share an institution, but the paper never says so; Figure 1 and Section III say 13 institutions and the introduction says 18 labs for the same collection effort. The affiliations list has 16 institutions among the authors.

Why it matters. It is easy to quote the wrong number; the headline facts (76 thousand, 564, 86, 52 buildings) are consistent throughout. Concept: One robot station · Collection protocol
F3. “36 thousand unique scenes” in a dataset with 564 scenes Numerical inconsistency

What the paper says. Appendix G provides camera-to-base calibrations for about 36 thousand unique scenes and a curated superset of 24 thousand scenes with both cameras calibrated.

What we found. The main text counts 564 unique scenes (merged from 2,080 identifiers); 36 thousand is 64 times that and fits better as a number of trajectories (47% of 76 thousand), as 24 thousand fits 32%. The appendix uses “scenes” for the unit that the main text calls episodes.

Why it matters. A reader looking for scene-level calibration statistics would draw the wrong conclusion; the data are valuable, the unit is mislabelled. Concept: Camera calibration · Definitions · example
F4. “An order of magnitude more scenes” holds against single-site datasets, not against the pooled one Numerical inconsistency

What the paper says. DROID has an order of magnitude more scenes than existing large robot manipulation datasets, and offers high diversity in tasks (verbs) and scenes.

What we found. Against the datasets the paper analyses (RT-1, RH20T, Bridge V2) the factors are 282, 80.6 and 23.5. Against Open X-Embodiment (311 scenes in Table I, “around 300” in the text) it is 1.8 times, against DobbE (216, a tool-based, non-robot dataset) 2.6 times; and DROID has 86 verbs against 217 for Open X-Embodiment. A “task” is a verb, so “86 tasks” is a count of verbs; Bridge V2 has 82.

Why it matters. DROID is much more diverse in scenes than any single dataset, and about as diverse in verbs as Bridge V2; it is not more varied than the whole pooled collection in tasks. Concept: The five axes · Definitions · example
F5. “The complete DROID dataset” is the first 40 thousand annotated episodes Numerical inconsistency

What the paper says. Policies are co-trained by mixing batches 50/50 with “the complete DROID dataset”, excluding unsuccessful trajectories.

What we found. Appendix F says the first 40 thousand successful trajectories with language annotations available at the time were used, and the scene-diversity subset of 7,362 is drawn from those. The Open X-Embodiment baseline has about 400 thousand trajectories, 10 times as many as the DROID subset; it is also described as having about 300 scenes (22 embodiments) in the main text and 311 in Table I.

Why it matters. The paper’s results concern a 40-thousand-episode slice; the full dataset is larger than anything tested, and the data-size comparison runs in DROID’s disfavour. Concept: Co-training · example

How far the claims go

F6. Same robot, same cameras: diversity and embodiment match are confounded Method caveat

What the paper says. Co-training with DROID improves performance and robustness more than co-training with the existing large Open X-Embodiment datasets, so DROID’s diversity is valuable.

What we found. All six tasks are run on the DROID hardware stack, the same arm, gripper, cameras and controller that produced DROID. Open X-Embodiment mixes about 22 other embodiments, some Franka, with other cameras and controllers, and the paper does not run an Open X-Embodiment subset restricted to this robot type or a DROID-style dataset from a single scene. In the scene-diversity test the 20-scene DROID subset (7 thousand trajectories, 12/30) is level with all of Open X-Embodiment (9/30), about 54 times larger, which is what an embodiment-match effect would produce as well as a scene-count effect.

Why it matters. The result supports “in-embodiment diverse data helps a lot”, not “DROID’s diversity beats Open X-Embodiment’s diversity”, and says little about robots other than the Franka arm. Concept: One robot station · Figure 8 · Figure 10
F7. The out-of-distribution gap to Open X-Embodiment is not clearly beyond chance Method caveat

What the paper says. DROID outperforms the next best method by 22% in distribution and by 17% out of distribution.

What we found. Each bar is 10 rollouts; the 17-point out-of-distribution gap is mostly against Open X-Embodiment (0.53 against 0.36). Out of distribution DROID beats it in 5 of 7 settings, ties in 1 and loses in 1 (novel chips: 0.3 against 0.6), with a sign-test p of 0.219; pooled, 37 against 25 successes of 70 gives an exact p of 0.0609. The in-distribution gap (45/60 against 26/60, p of 0.0007) and the gaps to no co-training are solid. The error bars appear to be standard errors across tasks, not rollouts.

Why it matters. The first claim is well supported; the second is a good bet but a single run with 10 rollouts per setting cannot confirm it, and the ranking of Open X-Embodiment against no co-training in distribution (0.43 against 0.53) goes the other way from the out-of-distribution one. Concept: The set-up · Figure 8 · example · example
F8. The scene-diversity test cannot separate scenes from everything else Method caveat

What the paper says. Co-training on a subset with diverse scenes has higher out-of-distribution performance than on a subset with 20 scenes, suggesting scene diversity is a driving factor.

What we found. Three settings and 10 rollouts each give 18/30 against 12/30 (exact p of 0.196). The random subset’s scene count is not reported, and the 20 scenes with the most data are not a random 20: they are the scenes with 368 trajectories each, against 135 on average, which may also be the easiest or most repeated tasks. Tasks, objects, viewpoints and collectors differ between the subsets too, so “scene diversity” stands for all of them.

Why it matters. It fits the story and may well be true, but the comparison is one draw of one random subset on three tasks, so “especially its diversity” is a hypothesis. Concept: Figure 10 · example
F9. The diversity measures are the authors’ own and are not tied to outcomes Ambiguity

What the paper says. DROID significantly increases diversity in tasks, objects, scenes, viewpoints and interaction locations over existing large datasets.

What we found. Task diversity is the number of verbs after GPT-4 de-duplication; object and scene-type labels come from a parser and a vision model; the scene count is a manual merge of 2,080 identifiers that the authors call conservative; the dataset comparisons use manually checked counts for four datasets. Only one of the five axes (scenes) is tested against performance, and in one small experiment. Viewpoints (1,417) and interaction locations are shown by plots without numbers.

Why it matters. The measures are reasonable and openly described, but the claim that more of each axis gives better policies is only tested for scenes. Concept: The five axes · Definitions

What is not tested

F10. One policy class, one robot, no language test Limitation

What the paper says. Across the board DROID improves policy success rate while increasing robustness to scene changes like distractors or novel object instances.

What we found. The paper tests one policy type (a ResNet-50 diffusion policy with 128-pixel images), one robot, one set of 6 tasks that the authors collected, trained one policy per task. The policies take a language instruction, but each is trained on one task, so language following is not tested, and no vision-language-action model is used. Later work (OpenVLA) puts DROID in a mixture at 10% and reports that the accuracy of the action tokens on it stayed low, a hint that the benefit for large models is less simple.

Why it matters. DROID is shown to be a useful ingredient for a small imitation learner on this robot; it is not shown to help generalist or vision-language-action models, or other robots. Concept: The policy · The set-up
F11. Smoothness and precision are claimed, not measured Limitation

What the paper says. Policies that leverage DROID are notably smoother and more precise than other comparisons, and DROID is the only method able to consistently finish all three Cook Lentils steps.

What we found. The evidence is one rollout per method in Figure 9 and the success counts of Figure 8. For Cook Lentils out of distribution the DROID policy scores 6/10 against 0/10 for Open X-Embodiment and 3/10 for no co-training, and in distribution 7/10 against 1/10 and 1/10; those counts do support the “only DROID finishes” statement, while smoothness and precision are not quantified (no path length, jerk or error measure).

Why it matters. The success counts are the result; the qualitative adjectives are impressions. Concept: Qualitative examples · Figure 8
F12. “In the wild” means research buildings more than homes Ambiguity

What the paper says. DROID is an “in-the-wild” dataset collected across 564 scenes in 52 buildings, with scenes ranging from offices to households.

What we found. Read by eye from Figure 4, about 36% of the scenes are industrial offices, 12% industrial kitchens and 12% offices, and the home-type rooms together come to about 40%, with the home kitchen the largest of them. The 52 buildings include research labs, and all data collectors are researchers and students at the participating institutions.

Why it matters. The scenes are much more varied than lab benches, but “in the wild” should be read as “outside the usual robot lab bench”, not as a sample of where service robots will work. Concept: The five axes · example

Knowledge check

Try each question first, then reveal the explanation.

Question 1

In your own words: how does the DROID protocol make a dataset diverse?

Explanation

The same robot station is used everywhere, the collector moves to a new scene about every 20 minutes, lists the feasible tasks, and the interface draws tasks at random and prompts for scene changes such as moved cameras, lighting changes and added objects.

Question 2

What do the co-training experiments show, and what do they not separate?

Explanation

Half-batches of DROID clearly improve a diffusion policy on six tasks over task data alone. They do not separate scene diversity from matching the robot, because the tasks run on DROID’s own hardware while Open X-Embodiment mixes many robots.

Question 3

What does “86 tasks” mean in the paper?

Explanation

It counts distinct verbs after de-duplication, not tasks as verb-and-object pairs; Bridge V2 has 82 verbs, and Open X-Embodiment 217.

Question 4

Name one place where the paper’s counts disagree.

Explanation

For example: 13 institutions and 18 labs; scene types counted as 9, 10, 11 or 12 in different places; or “36 thousand unique scenes” in the calibration appendix against 564 unique scenes in the main text.

Glossary

DROID: Distributed Robot Interaction Dataset: 76 thousand successful robot demonstrations from many scenes on one robot type.
In the wild: Outside a fixed laboratory setup; here, many rooms and buildings.
Teleoperation: A person controls the robot directly while data are recorded, here with a virtual-reality headset’s controllers.
Scene: A robot workspace; in this paper a new scene means a substantial change of the workspace.
Verb (as task): The paper counts distinct verbs in the instructions as the number of tasks.
Scene augmentation: A change the interface asks for during collection, such as moving the cameras or the lights.
Extrinsic calibration: Where a camera is, relative to the robot or another camera.
Proprioception: The robot’s own readings of its joints and gripper.
Diffusion policy: An imitation learner that generates a chunk of actions by removing noise in several steps.
Action chunk: A short sequence of future actions predicted together; here 16 predicted, 8 executed.
Co-training: Training on a mixture of two datasets; here every batch is half task data and half DROID or Open X-Embodiment.
Open X-Embodiment (OXE): A pooled collection of robot datasets from many labs and robot types.
In-distribution: Tested in the conditions of the task demonstrations, with some noise in the start positions.
Out-of-distribution (OOD): Tested with distractors, new objects or a camera shift that the task data did not contain.
Rollout: One attempt by the policy at a task; success rates here are out of 10 per setting.
Standard error: How much a measured rate would typically move by chance if the trials were repeated.
Sign test: A test that counts in how many settings one method beats another, ignoring by how much.
GPT-4V: A vision-language model used to label the type of each scene.
Polymetis: The robot control software used on the Franka arm.

References

Numbered notes in the text, like ^[1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., et al. (2024). DROID: A large-scale in-the-wild robot manipulation dataset. arXiv (v2, April 2025). arXiv:2403.12945
Cited for:
- 76 thousand successful demonstration trajectories, about 350 hours, collected in 564 scenes in 52 buildings with 86 distinct task verbs by 50 data collectors at 13 institutions over 12 months; each episode has three synchronised camera streams, camera calibration, depth and one to three natural-language instructions. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
- All data are collected on one shared robot platform: a Franka Panda arm with a Robotiq gripper on a height-adjustable desk, two adjustable external stereo cameras and a wrist stereo camera, teleoperated with a Meta Quest 2 headset at 15 Hz, replicated 18 times; a collection interface draws tasks at random and prompts scene changes about every 20 minutes. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
- Training a diffusion policy per task with half of every batch from DROID gives 0.75 success in distribution and 0.53 out of distribution over six tasks (10 rollouts per setting), against 0.53 and 0.17 without co-training and 0.43 and 0.36 with Open X-Embodiment; a 7,362-trajectory subset with diverse scenes scores 0.60 against 0.40 for one from 20 scenes on three out-of-distribution settings. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.

Go further

All paper guides · Back to the VLA track

How to use this page

Start here: the paper at a glance

The problem

The main contribution

How the dataset and the test work, in one paragraph

What it learns from, and what it is tested on

What the paper reports

What to keep in mind

Understand the method

1. One robot station, built many times

2. A collection protocol that forces variety

3. What is recorded, and the labels added afterwards

4. Fixing camera calibration after the fact

5. The policy used to test the data

Understand the data and the training

What “diverse” means: five axes and how they are measured

Definitions that decide the numbers: task, scene, success

Using it: 50/50 co-training

Understand the experiments

The set-up: six tasks, in-distribution and out-of-distribution

Does DROID improve performance and robustness? (Figure 8)

How important is scene diversity? (Figure 10)

What the qualitative examples add (Figure 9)

What the results show

What they do not show

Engineering deep dive

Critical paper review

Numbers that do not agree

F1. Four different lists of scene types Numerical inconsistency

F2. Institutions, labs and robots: 13, 18 and 18 Numerical inconsistency

F3. “36 thousand unique scenes” in a dataset with 564 scenes Numerical inconsistency

F4. “An order of magnitude more scenes” holds against single-site datasets, not against the pooled one Numerical inconsistency

F5. “The complete DROID dataset” is the first 40 thousand annotated episodes Numerical inconsistency

How far the claims go

F6. Same robot, same cameras: diversity and embodiment match are confounded Method caveat

F7. The out-of-distribution gap to Open X-Embodiment is not clearly beyond chance Method caveat

F8. The scene-diversity test cannot separate scenes from everything else Method caveat

F9. The diversity measures are the authors’ own and are not tied to outcomes Ambiguity

What is not tested

F10. One policy class, one robot, no language test Limitation

F11. Smoothness and precision are claimed, not measured Limitation

F12. “In the wild” means research buildings more than homes Ambiguity

Knowledge check

Question 1

Question 2

Question 3

Question 4

Glossary

References

Go further