5.1 — RT-1

Learning objectives

After this lesson, you should be able to:

• Describe RT-1’s inputs, outputs and main components.
• State the scale of its training data.
• Explain why data diversity, not only size, was studied.
• Place RT-1 relative to later VLAs.

The one-sentence idea

RT-1 shows a single language-conditioned policy trained on a large, varied real-robot dataset.

Variety of tasks, not only the number of frames.

What RT-1 was trying to show

Language and vision models became strong by training on huge, varied datasets. The RT-1 authors asked whether robots could do the same: large, diverse, task-agnostic datasets could help robots generalise.^[1] RT-1 (Robotics Transformer 1) is their answer: a Transformer-based model that encodes high-dimensional inputs and outputs, including camera images, instructions and motor commands, into compact tokens, which allows efficient inference for real-time control.^[1]

RT-1 is not a VLA by the strict definition (it was not built on a pretrained vision-language model; RT-2, lesson 5.2, did that), but it is the model that set up the recipe: one policy, language-conditioned, trained by imitation on a lot of real-robot data.

The recipe, from the sources

From Google’s own description: RT-1 combines a FiLM-conditioned EfficientNet, a TokenLearner and a Transformer.^[1] FiLM is the way the instruction steers the image processing (lesson 2.4). Each action dimension is discretised into 256 bins.^[1] The model has about 35 million parameters and runs at 3 Hz.^[1] It was trained on about 130,000 episodes covering more than 700 tasks, collected by 13 robots over 17 months.^[1]

episodes = 130_000
robots = 13
months = 17

per_robot = episodes / robots
print("episodes per robot:", round(per_robot))
print("episodes per robot per month:", round(per_robot / months))
print("episodes per robot per day (30-day months):", round(per_robot / (months * 30), 1))
print("episodes per task if spread evenly over 700 tasks:", round(episodes / 700))

episodes per robot: 10000
episodes per robot per month: 588
episodes per robot per day (30-day months): 19.6
episodes per task if spread evenly over 700 tasks: 186

Toy model. Plain arithmetic on the published totals, assuming even spreading. Real collection is uneven across robots, days and tasks.

About 20 episodes per robot per day, every day, for a year and a half. This is what “large-scale real-robot data” costs, and why lesson 3.2 says to think of episodes rather than frames.

Why the study measured data diversity

The authors studied how models generalise as a function of data size, model size and data diversity.^[1] Diversity matters because a policy is tested on cases it has not seen. A small experiment shows the shape of the effect: the same number of demonstrations, spread over few or many different goals, and then tested on goals that were never demonstrated:

import numpy as np
rng = np.random.default_rng(0)

def demo(goal):
    hand = rng.uniform(-1, 1, 2)
    return np.concatenate([hand, goal]), 0.3 * (goal - hand)         # observation (hand, goal) -> action

def error_on_new_goals(n_goals, total_demos=500):
    goals = rng.uniform(-1, 1, (n_goals, 2))
    data = [demo(goals[i % n_goals]) for i in range(total_demos)]
    X = np.array([d[0] for d in data]); Y = np.array([d[1] for d in data])
    errs = []
    for _ in range(300):                                               # test on goals never seen
        x, y = demo(rng.uniform(-1, 1, 2))
        j = np.argmin(np.linalg.norm(X - x, axis=1))                    # a nearest-neighbour "policy"
        errs.append(np.linalg.norm(Y[j] - y))
    return float(np.mean(errs))

for g in (2, 10, 50, 250):
    print(f"{g:>3} different goals, 500 demonstrations in total -> error on new goals {error_on_new_goals(g):.3f}")

  2 different goals, 500 demonstrations in total -> error on new goals 0.251
 10 different goals, 500 demonstrations in total -> error on new goals 0.105
 50 different goals, 500 demonstrations in total -> error on new goals 0.082
250 different goals, 500 demonstrations in total -> error on new goals 0.080

Toy model. A nearest-neighbour policy on a reaching task. It illustrates why variety, not only quantity, drives generalisation; it does not reproduce RT-1’s results.

The error falls as the same number of demonstrations covers more goals. That is the logic behind collecting 700+ tasks instead of recording one task many times.

What RT-1 left behind

Three ideas carried forward: tokenised actions (256 bins per dimension, reused by RT-2 and OpenVLA^[2]), language conditioning of the image processing through FiLM, and the habit of judging a robot model by how it generalises, not only by how it fits. Its 3 Hz rate is also a reminder of lesson 1.1: a decision about every third of a second means the world moves in between.

Common misconceptions

Engineering takeaways

1. About 130,000 episodes, 700+ tasks, 13 robots, 17 months.
2. 256 bins per action dimension.
3. About 35 million parameters, 3 Hz.
4. Diversity matters.

RT-1 asked: what if robots had their own big dataset?

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: RT-2, which reuses a pretrained vision-language model to output actions.

Back to the VLA roadmap