6.5 — RDT-1B

Learning objectives

After this lesson, you should be able to:

• Describe RDT-1B’s model and its target problem.
• Explain why two arms increase multimodality.
• Explain a unified action layout with a validity mask.
• State the cost of diffusion-based policies.

The one-sentence idea

A scaled diffusion transformer with a shared action layout for different robots.

Unified, interpretable action slots let data from different robots line up.

A diffusion model for two arms

RDT-1B (Robotics Diffusion Transformer) is a diffusion model with a scalable Transformer that handles heterogeneous multi-modal inputs, scaled to 1.2 billion parameters, which the authors describe as the largest diffusion-based foundation model for robotic manipulation.^[1] It targets bimanual (two-arm) manipulation, and it was fine-tuned on a self-created multi-task bimanual dataset of over 6,000 episodes. The authors report zero-shot generalisation to unseen objects and scenes, language instruction following, and learning new skills from about 1 to 5 demonstrations.^[1]

It is the diffusion counterpart to the flow-matching π0 (lesson 6.3) and the token-based OpenVLA (lesson 6.1): the generative method of lesson 4.6, scaled up. Two ideas make it interesting.

Why two arms make ambiguity worse

With one arm there are already several valid ways to do a task (lesson 3.4). With two arms the options multiply: either arm can grasp first, and each can approach from several sides. A model that averages demonstrations ends up with a motion that no demonstrator performed. A tiny illustration: some demonstrators move the left arm first, others the right arm first.

import numpy as np

# first-step action = [left arm movement, right arm movement]
left_first  = np.array([0.10, 0.00])
right_first = np.array([0.00, 0.10])
demos = np.array([left_first] * 50 + [right_first] * 50)

average = demos.mean(axis=0)
print("regression to the average:", np.round(average, 3).tolist(), "-> both arms creep forward by half the intended amount")
print("distance of that average from the nearest real demonstration:",
      round(float(min(np.linalg.norm(average - left_first), np.linalg.norm(average - right_first))), 3))

regression to the average: [0.05, 0.05] -> both arms creep forward by half the intended amount
distance of that average from the nearest real demonstration: 0.071

Toy model. Two exact demonstration modes in a two-number action. A diffusion policy samples one mode instead of averaging, as lesson 4.6 showed.

The abstract makes this motivation explicit in its framing of bimanual manipulation, with its multi-modal action distributions and scarce data; diffusion is the response to the first problem.

A unified action space

Robots have different numbers of joints and different units. RDT's answer for data scarcity is a physically interpretable unified action space: it represents different robots’ actions in one format while preserving the physical meaning of each original action, to help transfer physical knowledge across robots.^[1] In code, the idea looks like a fixed table of named slots that each robot fills in where it can, plus a mask that says which slots are real:

import numpy as np

slots = ["left_j1", "left_j2", "left_j3", "left_gripper", "right_j1", "right_j2", "right_j3", "right_gripper"]

def to_unified(robot_action):
    vec = np.zeros(len(slots)); mask = np.zeros(len(slots), dtype=bool)
    for name, value in robot_action.items():
        vec[slots.index(name)] = value; mask[slots.index(name)] = True
    return vec, mask

single_arm = {"right_j1": 0.2, "right_j2": -0.1, "right_j3": 0.4, "right_gripper": 1.0}
two_arms = {"left_j1": 0.1, "left_j2": 0.0, "left_j3": -0.2, "left_gripper": 0.0, "right_j1": 0.3, "right_j2": 0.2, "right_j3": 0.1, "right_gripper": 1.0}

for name, a in (("single-arm robot", single_arm), ("two-arm robot", two_arms)):
    vec, mask = to_unified(a)
    print(f"{name:<17} -> {vec.tolist()}")
    print(f"{'':<17}    valid slots: {int(mask.sum())} of {len(slots)}")

single-arm robot  -> [0.0, 0.0, 0.0, 0.0, 0.2, -0.1, 0.4, 1.0]
                     valid slots: 4 of 8
two-arm robot     -> [0.1, 0.0, -0.2, 0.0, 0.3, 0.2, 0.1, 1.0]
                     valid slots: 8 of 8

Toy model. A small hand-made layout. RDT defines its own, larger one; the point is that every number keeps a physical meaning and a place, so data from different robots line up.

With a shared layout and a mask, the loss is computed only on the slots that exist for that robot, and the model learns each slot’s meaning across all the robots that have it. This is the same action-definition discipline as lesson 4.1, applied to pooling.

The price

A 1.2-billion-parameter diffusion model needs many network passes to produce each action chunk (lesson 4.6), so latency is the cost. That is the main reason to compare it with token and flow approaches in lesson 6.6, rather than treat any one as best.

Common misconceptions

Engineering takeaways

1. 1.2B diffusion transformer for bimanual tasks.
2. Fine-tuned on 6K+ episodes.
3. Unified slots plus a mask.
4. Cost: many network passes.

Same slots for every robot; mask what is missing.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: a side-by-side comparison of all the models in this phase.

Back to the VLA roadmap