VLA · Paper guide
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
A humanoid-robot foundation model, explained through the questions an engineer would ask.
This is a guide to the GR00T N1 paper, not a sentence-by-sentence paraphrase. Part A is a 4-minute overview. Parts B to D explain the architecture, the training and the experiments, one idea at a time. Equations and code are folded away in Part E, and Part F is a critical review of the paper’s own numbers and claims.
How to use this page
- In a hurry: read Part A (about 4 minutes).
- To understand the model: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example with a humanoid picking up a bottle, then technical details and where to find them in the paper.
- To check or rebuild it: open the folded sections (equations, tensor shapes, hyperparameters, runnable examples) and read Part F.
Open the paper (arXiv:2503.14734) beside this page; section, figure and table numbers refer to it. We paraphrase and never reproduce its text. Three kinds of statement are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own code and never present results from NVIDIA; they are marked Our calculation when they only do arithmetic on the paper’s numbers.
Part A · about 4 minutes
Start here: the paper at a glance
The problem
A robot that can do many jobs needs a “mind” that turns what it sees and what it is told into movement. Language and image models learned this kind of breadth from enormous amounts of internet data. Robots have no such pile of data. Each robot design has little data of its own, and robots differ so much in bodies, sensors and controls that data from different robots does not simply add up.
GR00T N1 is NVIDIA’s attempt at a foundation model (one large model trained on broad data, then adapted) for humanoid robots (robots with a human-like body).
The main contribution
Two ideas, plus a release:
- A two-part model. A slower part that understands the scene and the instruction, and a faster part that produces the motor commands. They are trained together.
- A “data pyramid”. Training mixes real robot recordings (the scarce top), simulated and generated data (the middle), and human videos (the wide base). Data without robot commands is given estimated ones, so everything can be used by one training recipe.
- An open release of a 2.2-billion-parameter checkpoint, training data and simulation benchmarks.[1]
How the model works, in one paragraph
Camera images and a sentence like “pick up the bottle” go into a pretrained vision-language model (“System 2”), which turns them into a compact description of the situation. A second network (“System 1”) takes that description plus the robot’s own joint readings and produces a short sequence of 16 motor commands by starting from random noise and cleaning it up in four small steps. Small adapter networks, one per robot design, translate between each robot’s own numbers and the shared network.[1]
What it learns from
About 8,400 hours in total across four groups: real robot recordings (about 3,300 h), human videos (about 2,500 h), simulation (about 1,700 h) and generated robot videos (about 830 h), plus the web data the vision-language model was originally trained on. The paper’s own robot is a Fourier GR-1 humanoid, but only about 88 of the robot hours come from it.
What the paper reports
The model is fine-tuned for each task set (“post-training”) and compared with two models trained from scratch. Numbers below are as printed in the paper.
| Test | Baseline | GR00T N1 |
|---|---|---|
| Simulation, average of 3 benchmarks, 100 demonstrations per task (Table 2) | Diffusion Policy 33.4% | 45.0% |
| Real GR-1 humanoid, all training data (Table 3) | Diffusion Policy 46.4% | 76.8% |
| Real GR-1 humanoid, 10% of the data (Table 3) | Diffusion Policy 10.2% | 42.6% |
| Pre-trained model on two real tasks, no fine-tuning (§4.4) | no baseline run | 76.6% and 73.3% (15 trials each) |
What to keep in mind
- The comparison is between a large pre-trained model and baselines trained from scratch, so it cannot separate “better design” from “more data and pre-training”.
- Real-robot results use about ten trials per task. Differences of a few points are within noise.
- The authors list their own limits: short tabletop tasks only, no long tasks that combine walking with manipulation, and generated data that is not always physically accurate.
- Part F lists the places where the paper’s own numbers or wording disagree.
If you read nothing else: GR00T N1 shows that one recipe, a slow understanding module plus a fast action module trained on a pyramid of mixed data, gives strong results when fine-tuned from little robot data. It does not yet show which ingredient does the work.
Check your understanding: Why does the paper need a “data pyramid” at all?
Because there is no internet-scale pile of humanoid robot data. Real robot data is scarce and expensive; human video and simulation are plentiful but lack robot commands or differ from the real robot. The pyramid combines them so each kind covers what the others lack.
Part B
Understand the architecture
Follow one decision through the model: a humanoid is told “pick up the bottle”. We go block by block, plain words first, technical names second. No equations until Part E.
1. What goes in: images, an instruction, the robot’s own state
- Intuition
- Before deciding anything, a person looks at the table, remembers what they were asked, and feels where their hands are. The model gets the same three kinds of information.
- How it works
- Camera images and the instruction are given to the vision-language model. The robot’s own readings (joint angles, called proprioception) go straight to the action part. Each camera image is resized to 224 × 224 pixels and turned into 64 image tokens (small vectors, the pieces a transformer reads).
- Example
- Illustration, not from the paper A GR-1 humanoid sees a bottle on a table through its head camera. The instruction is “pick up the bottle”. Its arm and finger joints report their current angles.
- Technical details
- The 64 tokens come from a step called “pixel shuffle”, which merges neighbouring patches so there are fewer, wider tokens. The paper does not state the image encoder’s patch size, so the toy below uses an example grid.
Run the example: pixel shuffle
Toy illustrationWhat each part does
- Build a 16 × 16 grid of patch vectors, each with 6 numbers (invented sizes).
- Group every 2 × 2 block of neighbouring patches into one token by reshaping.
- Print the token counts and check that the first new token is the corner block.
import numpy as np grid, width = 16, 6 # an example: a 16 x 16 grid of patch embeddings, each of width 6 patches = np.arange(grid * grid * width, dtype=float).reshape(grid, grid, width) r = 2 # merge each 2 x 2 block of neighbouring patches into one token tokens = patches.reshape(grid // r, r, grid // r, r, width).transpose(0, 2, 1, 3, 4).reshape((grid // r) ** 2, r * r * width) print("before:", grid * grid, "tokens of width", width) print("after: ", tokens.shape[0], "tokens of width", tokens.shape[1]) print("first new token is the 2 x 2 block in the corner:", bool(np.array_equal(tokens[0], np.concatenate([patches[0, 0], patches[0, 1], patches[1, 0], patches[1, 1]]))))Output:
before: 256 tokens of width 6 after: 64 tokens of width 24 first new token is the 2 x 2 block in the corner: True
- Evidence in the paper
- §2.1 “Vision-Language Module (System 2)” · Figure 3
Check your understanding: Why does the robot’s joint state skip the vision-language model?
The state is a short list of numbers about the robot itself, not something the vision-language model was trained to read. In the paper’s Figure 3 it goes directly to the action module through a per-robot encoder.
2. System 2: the part that understands (Eagle-2)
- Intuition
- System 2 is the “what is going on and what am I supposed to do?” part. It is slow and deliberate, like reading a scene carefully.
- The problem it solves
- Telling a bottle from a cup, finding “the one on the left”, and understanding the words all need broad knowledge about the world. Robot data alone is too small to teach that.
- How it works
- The authors use Eagle-2, a vision-language model (VLM) already trained on internet-scale image and text data. It combines an image encoder (SigLIP-2) with a language model (SmolLM2).[2] The image tokens and the instruction are read together. The output the robot uses is not a sentence: it is the internal vectors (“embeddings”) at one layer of the language model.
- Example
- Illustration, not from the paper For “pick up the bottle”, the vectors at that layer carry the model’s combined reading of the picture and the words: roughly which object, where it is and what is wanted.
- Why the authors chose it
- Starting from a pretrained VLM brings in its general understanding. Taking a middle layer rather than the last one was an empirical choice: the authors report it ran faster and gave a higher success rate. For the released 2B model they use the 12th layer. The paper gives no numbers for that comparison.
- Technical details
- The VLM is run at about 10 Hz on an NVIDIA L40 GPU (one update every 100 ms). The features have shape batch × sequence length × hidden size. Eagle-2 itself is described in another paper and is not re-explained. The text tokenizer is frozen; the vision encoder is trainable (Table 6).
- Evidence in the paper
- §1 (System 2 at 10 Hz) · §2.1 · Table 6
Check your understanding: The robot uses embeddings from the 12th layer instead of generated text. Why might that be useful?
Embeddings keep much more information than a short sentence and skip the later layers, which saves time. The paper reports faster inference and better success than using the final layer, without giving the numbers.
3. System 1: the part that acts (a Diffusion Transformer with flow matching)
- Intuition
- System 1 is the “muscle memory” part. It is fast. Think of sculpting: start with a rough lump (random noise) and make a few quick corrections until it becomes the movement you want.
- The problem it solves
- The robot must output smooth, continuous motor commands, many per second, and keep up with a real-time control loop.
- How it works
- The model starts with a random action sequence and improves it in four small steps. At every step it looks at the vision-language tokens from System 2 (through cross-attention, where one sequence of tokens “looks at” another) and at the robot’s state. The result is a clean sequence of motor commands.
- Example
- Illustration, not from the paper Step 1: random wiggles. Step 2: the wiggles start to look like a reach. Step 3: they bend toward the bottle’s location. Step 4: a clean reaching motion.
- Why the authors chose it
- Generating whole action sequences by gradual denoising is how recent robot policies such as Diffusion Policy and π0 work, and cross-attention keeps the connection to the vision-language model simple. The paper contrasts this with π0’s mixture-of-experts bridge but runs no test swapping the two.
- Technical details
- The network is a Diffusion Transformer (DiT) trained with flow matching, a training method that teaches the network which direction to move a noisy action toward a clean one. Its layers alternate between self-attention over the action tokens and cross-attention to the vision-language tokens, and are told the current denoising step through adaptive layer normalisation.[3] The equations, the sign question and a runnable test are in Part E.[4] See also lesson 4.7 and lesson 4.6.
- Evidence in the paper
- §1 (System 1 at 120 Hz) · §2.1 “Diffusion Transformer Module” · Figure 3 · Equation 1
Check your understanding: Why four steps and not one?
A single step has to jump from noise to the final answer in one move, which is hard. A few small corrections follow the learned directions more accurately. The paper reports that four steps worked well; in our toy test one step fails and four work.
4. Per-robot adapters: one shared brain, many bodies
- Intuition
- Different robots speak different “languages”: a single arm has a handful of numbers, a humanoid with hands has dozens. The adapters are interpreters that translate each robot’s numbers into the shared language and back.
- The problem it solves
- The shared network expects inputs and outputs of one fixed width, but different robots have different numbers of joints, sensors and control modes. Without translation the data from different robots cannot be mixed.
- How it works
- Every robot design gets its own small network (a multi-layer perceptron, MLP: a few simple layers) that turns its state into the shared size, and another that does the same for its actions. After the last DiT block, a third per-robot network turns the result back into that robot’s actions. The action encoder also receives the denoising step, so the network knows how noisy its input is.
- Example
- Illustration, not from the paper A single-arm robot with 7 joints and a humanoid with 29 values both end up as the same shape inside the DiT, then come back out in their own sizes.
- Technical details
Run the example: one encoder and decoder per robot
Toy illustrationWhat each part does
- Invent three robots with different state and action sizes.
- For each, create random matrices standing in for the trained encoder and decoder.
- Send a state and a noisy action chunk through; print the shared shape and the shape that comes back.
import numpy as np rng = np.random.default_rng(0) embed_dim = 8 # the shared size the DiT works in (the real one is much larger) robots = {"arm A": (7, 7), "humanoid B": (29, 29), "gripper C": (4, 5)} # (state size, action size): invented examples class RobotModules: def __init__(self, state_dim, action_dim): self.W_state = rng.normal(size=(state_dim, embed_dim)) / np.sqrt(state_dim) # state encoder self.W_action = rng.normal(size=(action_dim, embed_dim)) / np.sqrt(action_dim) # action encoder self.W_out = rng.normal(size=(embed_dim, action_dim)) / np.sqrt(embed_dim) # action decoder modules = {name: RobotModules(s, a) for name, (s, a) in robots.items()} H = 16 for name, (s, a) in robots.items(): state = rng.normal(size=s); noisy_chunk = rng.normal(size=(H, a)) state_token = state @ modules[name].W_state # 1 token of width embed_dim action_tokens = noisy_chunk @ modules[name].W_action # H tokens of width embed_dim dit_input = np.vstack([state_token[None, :], action_tokens]) # what the shared DiT sees: (1 + H) x embed_dim, for every robot decoded = dit_input[1:] @ modules[name].W_out # back to this robot's action size print(f"{name:<11} state {s:>2}, action {a:>2} -> DiT input {dit_input.shape} -> decoded chunk {decoded.shape}")Output:
arm A state 7, action 7 -> DiT input (17, 8) -> decoded chunk (16, 7) humanoid B state 29, action 29 -> DiT input (17, 8) -> decoded chunk (16, 29) gripper C state 4, action 5 -> DiT input (17, 8) -> decoded chunk (16, 5)
Random matrices and made-up sizes. In the paper these are trained networks; the point is only that every robot is translated into the same shape on the way in and back out.
- Evidence in the paper
- §1 · §2.1 “State and Action Encoders” · Figure 3
Check your understanding: What happens when a robot with a different number of joints is added?
It gets its own encoder and decoder; the shared network stays the same. In the paper’s post-training the model is then fine-tuned on one robot’s data at a time.
5. Action chunks and execution
- Intuition
- Instead of choosing one tiny movement at a time, the robot plans a short stretch of movement, such as the next part of a reach, and carries it out while looking again.
- The problem it solves
- Deciding every single step from scratch is slow and makes motion jittery. Planning a short sequence is smoother and lets the slow and fast parts run at different speeds.
- How it works
- Each time System 1 runs, it produces a chunk of H = 16 consecutive actions. System 2 refreshes its description of the scene about ten times a second, while System 1 produces up to 120 actions a second. So several actions share one description of the scene.
- Example
- Illustration, not from the paper A reach toward the bottle is planned as 16 small commands, sent one after another while the next camera frames are processed.
- Why the authors chose it
- Chunking follows ACT (Action Chunking with Transformers) and other chunked policies: smoother motion and less often running the heavy model.[5] See lesson 4.4.
- Technical details
- The paper reports that sampling one chunk takes 63.9 ms on an L40 in bf16 (a compact 16-bit number format). It does not say whether this includes the vision-language pass or how chunks are scheduled to give a steady 120 Hz stream, and its teleoperation data was recorded at 20 Hz. The example shows what these numbers imply.
Run the example: parameters, memory and the 120 Hz claim
Our calculationWhat each part does
- Subtract the VLM’s 1.34 billion parameters from the 2.2 billion total.
- Convert parameters to memory at 2 bytes each (bf16).
- Turn 63.9 ms per chunk and 16 actions per chunk into actions per second, and compare with 120 Hz.
total_params, vlm_params = 2.2e9, 1.34e9 action_side = total_params - vlm_params print("parameters outside the VLM (DiT + per-robot encoders/decoders):", round(action_side / 1e9, 2), "billion =", round(100 * action_side / total_params), "% of the total") print("memory for the weights in bf16 (2 bytes each):", round(total_params * 2 / 1e9, 1), "GB") H = 16 # actions per chunk chunk_ms = 63.9 # time to sample one chunk on an L40 (paper) system1_hz = 120 # action rate stated in the introduction system2_hz = 10 # VLM rate stated in the introduction print("a chunk covers", round(H / system1_hz * 1000, 1), "ms of motion at", system1_hz, "Hz") print("chunks per second the GPU could sample:", round(1000 / chunk_ms, 1), "->", round(1000 / chunk_ms * H), "actions per second") print("chunks per second needed for a steady", system1_hz, "Hz stream:", system1_hz / H) print("share of the GPU's time that would take:", round(100 * (system1_hz / H) * chunk_ms / 1000), "%") print("actions produced per VLM update:", system1_hz // system2_hz)Output:
parameters outside the VLM (DiT + per-robot encoders/decoders): 0.86 billion = 39 % of the total memory for the weights in bf16 (2 bytes each): 4.4 GB a chunk covers 133.3 ms of motion at 120 Hz chunks per second the GPU could sample: 15.6 -> 250 actions per second chunks per second needed for a steady 120 Hz stream: 7.5 share of the GPU's time that would take: 48 % actions produced per VLM update: 12
If 63.9 ms is the full cost of a chunk, about 250 actions per second are possible, so a steady 120 Hz stream would use roughly half of the GPU’s time. This is arithmetic on the paper’s numbers, not a measurement. Part F lists what the paper leaves open.
- Evidence in the paper
- §1 · §2.1 “Action chunks”, “Model size and latency” · §3.1 (20 Hz teleoperation)
Check your understanding: If System 2 updates 10 times a second and System 1 produces 120 actions a second, how many actions share each scene description?
About 12 (120 ÷ 10). The fast part keeps acting on a description that can be up to about 100 ms old.
Part C
Understand the training
A robot model is only as good as its data. This part explains where GR00T N1’s data comes from, how data that has no robot commands is made usable, and how pre-training and post-training fit together.
The data pyramid
- Real robot datathe scarce top: about 3,289 hours in Table 7, only about 88 h on the paper’s own GR-1 robot
- Synthetic datathe middle: simulation (about 1,743 h) and generated robot videos (about 827 h)
- Human videos and web datathe wide base: about 2,517 h of human video, plus the web data the vision-language model was trained on
- Intuition
- Learning to cook: you watch many cooking videos (lots of data, but you cannot see exactly what the cook’s muscles did), practise in a safe kitchen simulator, and finally do a few real sessions in your own kitchen. Each stage adds something the others cannot.
- The problem it solves
- There is no internet-scale pile of humanoid robot data, and pooling data from different robots gives “islands” that do not fit together.
- How it works
- Training data is organised in three layers, from the wide base to the narrow top, and all layers are used together (“co-training”). The lower layers teach how things look and how people act; the upper ones ground the model in what a real robot does.
- Why the authors chose it
- Each layer fixes a different weakness: human video is plentiful but has no robot actions; simulation and generated video are cheap but imperfect; real robot data is accurate but expensive.
- Evidence in the paper
- Figure 1 · §1 · Table 1 (which technique applies to which source) · Table 7 (frames and hours per dataset)
Check your understanding: Is the pyramid’s shape a measurement of how many hours were used?
No. It is a picture of kinds of data. In Table 7 the real-robot hours (about 3,289) actually exceed the human-video hours (about 2,517), because the “wide base” also includes web data that the table does not count.
Real robot demonstrations
- Intuition
- A person puppeteers the robot through a task while everything is recorded. The robot then learns to copy it.
- The problem it solves
- Real demonstrations are the most trustworthy data, because they show exactly what this kind of robot can do. But each one needs a human operator, so they do not scale.
- How it works
- Three sources: (1) the authors’ own GR-1 humanoid recordings, made with wrist trackers and finger-tracking gloves and converted into robot joint commands (“retargeting”) at 20 Hz; (2) seven datasets from Open X-Embodiment (RT-1, Bridge-v2, Language Table, DROID, MUTEX, RoboSet, Plex);[6] (3) the 140,000 trajectories of AgiBot-World available when training started.
- Example
- Illustration, not from the paper An operator wearing tracking gloves picks up a bottle; their hand motion is converted into the humanoid’s arm and finger joint commands, and the head-camera video is stored with them.
- Why the authors chose it
- These give real actions and real physics. The paper’s Table 7 shows that most real hours come from other robots (AgiBot-Alpha alone is about 60% of the robot hours); the GR-1 data is the small part closest to the evaluation robot.
- Evidence in the paper
- §3.1 · Table 7 · Figure 6 (teleoperation set-up)
Check your understanding: Why is the GR-1 data only a small share of the robot hours?
Teleoperating a humanoid with two arms and dexterous hands is expensive; the authors collected about 88 hours themselves and relied on much larger datasets from other robots for the rest.
Human videos
- Intuition
- Watching people use their hands teaches what objects are for and how motion looks, even though you cannot read their muscle commands off the video.
- The problem it solves
- Human videos are plentiful and cheap, but they have no robot actions, so there is nothing for the model to imitate directly.
- How it works
- Seven public datasets are used (Ego4D, Ego-Exo4D, Assembly-101, EPIC-KITCHENS, HOI4D, HoloAssist, RH20T-Human). Their actions are estimated with latent actions, below.
- Why the authors chose it
- They add variety in objects, tasks and motion that robot data cannot match at this scale, and prior work (Ye et al.) reported positive transfer from human video with latent actions.
- Evidence in the paper
- §3.3 · Figure 14 · Table 7
Check your understanding: What is missing from human video that robot training needs?
The robot’s own commands (joint angles or hand motions in the robot’s terms). The paper’s remedy is to estimate stand-in actions from the video itself.
Latent actions: labelling video without a robot
- Intuition
- Show someone two photos of a hand, one before and one after, and ask “what happened between them?”. The answer, “it moved to the right”, is a kind of action label that you made up from the pictures alone.
- The problem it solves
- Human and generated videos have no action labels, but the training recipe needs an action for every example.
- How it works
- A small model looks at a frame and a later frame and writes a short summary of what changed (the latent action). A second part must rebuild the later frame from the first frame plus that summary, so the summary has to carry the information about the change. After training, only the part that writes the summary is kept and used as the label.
- Example
- Illustration, not from the paper Frame 1: a hand left of the bottle. Frame 2: the hand next to the bottle. The latent action is a compact code that means roughly “moved right”.
- Why the authors chose it
- It needs no robot, so it works on any video, and one shared model puts every robot and the humans in the same label space, which could help transfer between bodies (the paper’s wording is “potentially”). The label is treated as a separate “robot” named LAPA (latent action pretraining, after Ye et al.).
- Technical details
- The model is a VQ-VAE: its continuous summary is replaced by the nearest entry of a small learned codebook (vector quantisation). For GR00T N1 the training label is the continuous summary before quantisation. The method follows Ye et al.
Run the example: a toy latent action
Toy illustrationWhat each part does
- Make 2,000 pairs of “frames”, each just the position of a hand now and a few frames later.
- The “encoder” is the change between them. A two-entry codebook is fitted by a few k-means steps.
- Compare rebuilding the later position from the quantised code with rebuilding from the continuous value, and count how many distinct labels each gives.
import numpy as np rng = np.random.default_rng(0) # Frame pairs: where a hand is in the picture now, and a few frames later. There are no action labels. n = 2000 x_now = rng.uniform(0.2, 0.8, n) moves = rng.choice([-0.3, 0.3], n) * rng.uniform(0.7, 1.3, n) # the hand moves left or right, by varying amounts x_future = x_now + moves z = x_future - x_now # "encoder": the change it sees between the two frames codebook = np.array([-0.1, 0.1]) # vector quantisation: a codebook of 2 entries, a few k-means steps for _ in range(10): nearest = np.abs(z[:, None] - codebook[None, :]).argmin(axis=1) codebook = np.array([z[nearest == k].mean() for k in range(2)]) print("codebook entries:", np.round(codebook, 3).tolist()) print("rebuild error using the quantised code: ", round(float(np.abs(x_now + codebook[nearest] - x_future).mean()), 3)) print("rebuild error using the continuous value:", round(float(np.abs(x_now + z - x_future).mean()), 3)) print("distinct labels: quantised", len(np.unique(codebook[nearest])), "| continuous", len(np.unique(np.round(z, 6)))) print("what the two codes mean:", {("left" if c < 0 else "right"): int((codebook[nearest] == c).sum()) for c in codebook})Output:
codebook entries: [-0.3, 0.3] rebuild error using the quantised code: 0.046 rebuild error using the continuous value: 0.0 distinct labels: quantised 2 | continuous 1993 what the two codes mean: {'left': 1022, 'right': 978}A one-dimensional “video” and a two-entry codebook. The real model uses images and learned neural networks, but the logic is the same: a code for what changed, with the continuous version kept as the training label.
- Evidence in the paper
- §2.2 “Latent actions” · Figure 4 (similar latent actions retrieved across robots and humans; a qualitative picture, not a measured accuracy) · Figure 9 (latent-action labels against inverse-dynamics labels)
Check your understanding: Why keep the continuous label instead of the code from the codebook?
The code has only as many values as codebook entries (two in our toy), so it throws information away; the continuous value keeps the size of the change too.
Inverse dynamics: labelling video with a model trained on real robots
- Intuition
- If you know where the arm was in one photo and where it is in the next, you can work out what command moved it. An inverse dynamics model (IDM) does this: two pictures in, the actions between them out.
- The problem it solves
- Generated robot videos show a robot moving but come without its commands.
- How it works
- An IDM is trained on real robot data of one embodiment: given a current frame and a future frame, it produces the chunk of actions in between. It is then run on every step of the generated videos to produce pseudo-actions (estimates, not what a robot really did). The policy treats them as the actions of a separate “robot”.
- Why the authors chose it
- The pseudo-actions are expressed in the target robot’s own action space, unlike latent actions. The paper compares both label types in RoboCasa (Figure 9): latent labels are slightly better with 30 demonstrations and the IDM is better with 100 and 300. The authors explain this by the IDM getting better with more data; that explanation is not tested directly.
- Technical details
- The IDM uses the same Diffusion Transformer as System 1 with SigLIP-2 image embeddings and is trained with flow matching, for 30K or 60K steps per embodiment depending on dataset size. The authors found that adding state or more frames did not improve validation accuracy.
Run the example: detection loss and pseudo-label quality
Toy illustrationWhat each part does
- Compute the auxiliary object-location loss (a squared distance) for one example (the auxiliary loss described in Part E).
- Make a toy world where the next state is the state plus the action.
- Estimate the action from the two states with a noisier or cleaner “observation” and print the labelling error.
import numpy as np # the auxiliary loss: squared distance between predicted and true normalised box centres x_gt = np.array([0.62, 0.40]) # box centre / (width, height) x_pred = np.array([0.55, 0.47]) print("L_det =", round(float(np.sum((x_pred - x_gt) ** 2)), 4)) # toy IDM: true dynamics next = state + action; an IDM recovers action = next - state, with some error rng = np.random.default_rng(0) state = rng.normal(size=(1000, 2)) action = rng.normal(size=(1000, 2)) nxt = state + action for noise in (0.5, 0.1): obs_noise = rng.normal(scale=noise, size=nxt.shape) # a weaker IDM sees a noisier next image pseudo = nxt + obs_noise - state print(f"observation noise {noise}: mean pseudo-action error = {np.abs(pseudo - action).mean():.3f}")Output:
L_det = 0.0098 observation noise 0.5: mean pseudo-action error = 0.395 observation noise 0.1: mean pseudo-action error = 0.082
Invented numbers. It shows only that cleaner inputs give more accurate pseudo-labels, not how accurate the paper’s IDM is.
- Evidence in the paper
- §2.2 and §2.3 “Post-training with neural trajectories” · Appendix F (IDM) · Figure 9
Check your understanding: Are IDM pseudo-actions the same as the actions a real robot executed?
No. They are estimates inferred from two frames. The paper reports they help in its tests, but they carry the IDM’s errors.
Simulation data (DexMimicGen)
- Intuition
- Teach the robot one way to pick up a cup from a few human demonstrations; then a computer replays and adapts that motion for hundreds of thousands of different cup positions, keeping only the ones that work.
- The problem it solves
- Collecting real humanoid data at scale is expensive because both arms and dexterous hands must be controlled at once.
- How it works
- Start from a few dozen human demonstrations. Cut them into short segments, one per object. For a new scene, shift each segment to the object’s new position, smooth the join, and run it in simulation. Only runs that succeed are kept.
- Why the authors chose it
- It multiplies a small amount of human effort into a large dataset. The paper reports 780,000 simulated trajectories across pre- and post-training, said to equal 6,500 hours (nine continuous months) of human demonstration, generated in 11 hours.
- Technical details
- For pre-training, 54 pairs of source and target containers × 10,000 demonstrations each gives 540,000.Simulated data inherits the simulation-to-reality gap, which the paper acknowledges.[7] See lesson 7.7.
Run the example: the data-generation numbers
Our calculationWhat each part does
- Divide 827 hours of generated video by 88 hours of real teleoperation.
- Convert 6,500 hours over 780,000 trajectories into seconds per trajectory.
- Compare 6,500 hours with the 11 hours it took, and express 6,500 hours in months.
real_hours, neural_hours = 88, 827 print("neural / real teleoperation hours:", round(neural_hours / real_hours, 1), "times") trajectories, hours_equiv, wall_hours = 780_000, 6500, 11 print("average length of a simulated trajectory:", round(hours_equiv * 3600 / trajectories), "seconds") print("speed-up over human demonstration time:", round(hours_equiv / wall_hours), "times") print("6,500 hours as continuous months of 30 days:", round(hours_equiv / (24 * 30), 1))Output:
neural / real teleoperation hours: 9.4 times average length of a simulated trajectory: 30 seconds speed-up over human demonstration time: 591 times 6,500 hours as continuous months of 30 days: 9.0
Arithmetic on the paper’s numbers. Part F notes that the 780,000 cannot be itemised from the numbers the paper prints.
- Evidence in the paper
- §2.2 “Simulation trajectories” · §3.2 · Figure 6 · Table 1
Check your understanding: Why keep only successful runs?
A replayed motion that fails (for example the hand misses the object) would teach the wrong behaviour; filtering keeps the data quality high.
Neural trajectories: generated robot videos
- Intuition
- Show a video model the first frame of a scene and tell it “now pick up the other cup”. It imagines the whole video, as if a robot had done it.
- The problem it solves
- Even with simulation, covering many objects and instructions is costly; and some motions (pouring from a mesh cup, for example) are hard to simulate.
- How it works
- An image-to-video model (Wan 2.1, 14 billion parameters) is fine-tuned on the 88 hours of GR-1 teleoperation. It is then given existing first frames with new instructions, which another model writes (“pick up {object} from {place A} to {place B}”). A judge model watches eight frames of each video and checks it follows its instruction; videos that fail are re-captioned instead of dropped. All videos are then labelled with latent actions and IDM pseudo-actions.
- Why the authors chose it
- It adds quantity and variety of “what if” situations without anyone teleoperating the robot: the paper reports about 827 hours of video, roughly ten times the real GR-1 data. The 827 hours do not replace the real 88 hours; the videos only carry estimated actions.
- Technical details
- Training clips are 81 frames at 480P. Generation took about 105,000 L40 GPU-hours.
Run the example: neural trajectory numbers
Our calculationWhat each part does
- Spread 88.4 hours of real data over 3,000 training samples to estimate each sample’s length in seconds.
- Divide by 81 frames to get an effective frame rate.
- Turn “2 minutes of GPU time per second of video” into GPU-hours, and spread them over 3,600 GPUs.
hours_real, samples = 88.4, 3000 print("average length of a training sample if the 3,000 samples are the 88.4 hours:", round(hours_real * 3600 / samples), "seconds") print("each squeezed into 81 frames -> an effective", round(81 / (hours_real * 3600 / samples), 2), "frames per second") video_hours = 827 seconds_of_video = video_hours * 3600 gpu_hours = seconds_of_video * 2 / 60 # 2 minutes of GPU time per second of video print("GPU-hours from '2 minutes per second of video':", round(gpu_hours), "(paper: about 105,000)") print("105,000 GPU-hours on 3,600 GPUs takes", round(105_000 / 3600, 1), "hours =", round(105_000 / 3600 / 24, 2), "days (paper: ~1.5 days)")Output:
average length of a training sample if the 3,000 samples are the 88.4 hours: 106 seconds each squeezed into 81 frames -> an effective 0.76 frames per second GPU-hours from '2 minutes per second of video': 99240 (paper: about 105,000) 105,000 GPU-hours on 3,600 GPUs takes 29.2 hours = 1.22 days (paper: ~1.5 days)
Arithmetic on numbers printed in the paper, with the assumption stated in the first step.
- Evidence in the paper
- §2.2 “Neural trajectories” · Figure 5 and Figure 13 (examples) · §3.2 · Appendix F · Figure 9 (effect on success rates, Part D)
Check your understanding: What do generated videos add that simulation does not?
Scenes and motions that are hard to build in a simulator (liquids, flexible objects) and variety driven by language prompts. The cost is that the videos can be physically inaccurate, which the paper lists as a limitation.
Pre-training, co-training and post-training
- Intuition
- Pre-training is school: broad education from every kind of data. Post-training is on-the-job training: a short course on one robot and its tasks.
- The problem it solves
- A single broad model is not by itself good at one particular robot’s tasks, and fine-tuning data for that robot is scarce.
- How it works
- In pre-training the model learns from all layers of the pyramid with one loss (flow matching). Human videos use latent actions as the target; robot data uses both its real actions and latent labels; generated videos use latent and IDM labels. In post-training the model is fine-tuned on one robot’s data; the language part of the vision-language model stays frozen. For tasks with little data, generated videos are added, sampled 1:1 with real demonstrations.
- Example
- Illustration, not from the paper Pre-train on all the data; then fine-tune on 100 demonstrations of picking up bottles with one GR-1 robot.
- Why the authors chose it
- This follows the foundation-model recipe: learn broadly first, adapt cheaply later. See lesson 5.4 and lesson 7.4. The paper’s experiments in Part D are all post-training results except the two pre-trained-only tests.
- Technical details
- Pre-training took about 50,000 H100 GPU-hours on up to 1,024 GPUs; post-training is lighter (the paper tested fine-tuning on a single A6000 GPU). Hyperparameters are in Part E.
Run the example: what “1:1 sampling” means
Our calculationWhat each part does
- Fix 3,000 generated videos per task (the RoboCasa setting).
- For each number of real demonstrations, divide 3,000 by it.
- Print how many times more often each real demonstration is drawn than each generated one.
neural_per_task = 3000 # RoboCasa runs add 3,000 neural trajectories per task (Figure 9 caption) for real in (30, 100, 300): # demonstrations per task in the RoboCasa post-training regimes ratio = neural_per_task / real print(f"{real:>3} real demos + {neural_per_task} neural per task: each real demo is drawn about {ratio:>5.0f} times as often as each neural trajectory")Output:
30 real demos + 3000 neural per task: each real demo is drawn about 100 times as often as each neural trajectory 100 real demos + 3000 neural per task: each real demo is drawn about 30 times as often as each neural trajectory 300 real demos + 3000 neural per task: each real demo is drawn about 10 times as often as each neural trajectory
A 1:1 ratio is about draws, not counts. With few real demonstrations, each is revisited many times. The paper does not discuss overfitting from this.
- Evidence in the paper
- §2.3 · Table 6 · §1 (co-training)
Check your understanding: If a model is pre-trained on all robots, is the fine-tuned robot model still one shared set of weights?
No. After pre-training there is one model; for the experiments, it is fine-tuned separately for each robot, so each deployed model has its own weights.
Part D
Understand the experiments
What did the authors test, against what, and what do the numbers let us conclude? The tables below use the paper’s numbers; the notes after each table are ours.
The set-up: tasks, baselines and metrics
- Intuition
- To judge a new student you give them tests, compare them with other students, and decide how to score. All three choices affect what the grade means.
- How it works
Simulated tests (three benchmarks, each fine-tuned with 30, 100 or 300 demonstrations per task):
- RoboCasa Kitchen: 24 kitchen tasks (pick-and-place, doors, drawers, buttons, faucets) on a single Franka arm with three cameras.
- DexMimicGen suite: 9 two-armed tasks on three different robots (two with arms and grippers or hands, one humanoid).
- GR-1 Tabletop: 24 tasks on the humanoid with dexterous hands and a single head camera, built to mirror the real tasks. None of its container combinations appeared in pre-training.
Real-robot tests on the GR-1: 13 tasks in four groups (pick-and-place, articulated objects, industrial objects, two-robot coordination), each with 15 minutes to 3 hours of human demonstrations, each scored over about 10 trials with partial credit. A second regime trains on a random 10% of that data. Two further tests use the pre-trained model without fine-tuning (15 trials each).
Baselines: BC-Transformer (a behaviour-cloning transformer that reads the last 10 frames and predicts the next 10 actions) and Diffusion Policy (a denoising policy using a U-Net that reads one frame and outputs 16 actions). Both are trained from scratch: no pre-trained vision-language model, no cross-robot pre-training. On the real robot only Diffusion Policy is run.
Metric: success rate. In simulation it is averaged over 100 trials and the best score among the last 5 saved checkpoints is reported. On the real robot a trial can earn partial credit.
- Evidence in the paper
- §4.1 · §4.2 · §4.3 · Figures 7 and 8
Check your understanding: What does “trained from scratch” change about how to read the comparison?
The baselines never saw the large pre-training data or a pre-trained language model, so any gap mixes two things: GR00T’s design and its pre-training. The paper does not run an architecture-only comparison.
Simulation results
- Intuition
- Same exams, same amount of practice material: who scores higher?
- How it works
- Each model is fine-tuned on the same number of demonstrations per task and tested on new object placements.
- Technical details
Table 2 as printed: success rate (%). 100 demonstrations per task RoboCasa DexMG GR-1 Average BC-Transformer 26.3 53.9 16.1 26.4 Diffusion Policy 25.6 56.1 32.7 33.4 GR00T-N1-2B 32.1 66.5 50.0 45.0 The “Average” column is an average over all 57 tasks (24 + 9 + 24), not over the three columns. The widest gap is on the GR-1 humanoid benchmark: 50.0 against 32.7. Our calculation Table 4 (per task, every dataset size) is in the example below, which also finds a mismatch with Table 2 that Part F describes.
Run the example: check Table 2 and compare it with Table 4
Our calculationWhat each part does
- Type in Table 2 and the number of tasks in each benchmark.
- Recompute each “Average” as a plain mean and as a task-weighted mean, and compare with the printed value.
- Recompute the 100-demonstration averages from Table 4 and compare the DexMG numbers.
tasks = {"RoboCasa": 24, "DexMG": 9, "GR-1": 24} table2 = { # 100 demonstrations per task, as printed in Table 2: RoboCasa, DexMG, GR-1, Average "BC-Transformer": ([26.3, 53.9, 16.1], 26.4), "Diffusion Policy": ([25.6, 56.1, 32.7], 33.4), "GR00T-N1-2B": ([32.1, 66.5, 50.0], 45.0), } n = list(tasks.values()) for name, (cols, stated) in table2.items(): plain = sum(cols) / 3 weighted = sum(c * k for c, k in zip(cols, n)) / sum(n) print(f"{name:<17} plain mean of the 3 columns {plain:5.1f} | mean weighted by task count {weighted:5.1f} | stated {stated}") print("-> the 'Average' column is the average over all 57 tasks, not over the 3 benchmarks (lesson 8.1: micro vs macro).") print("GR-1 gap, GR00T minus Diffusion Policy:", round(50.0 - 32.7, 1), "points") # Table 4 / Figure 10 give different DexMG numbers at 100 demonstrations: table4_at_100 = {"Diffusion Policy": [25.6, 46.9, 32.7], "GR00T-N1-2B": [32.1, 58.5, 50.0]} print() for name, cols in table4_at_100.items(): print(f"{name:<17} Table 4 / Figure 10 at 100 demos: {cols} -> weighted average {sum(c * k for c, k in zip(cols, n)) / sum(n):.1f}") print("DexMG at 100 demos: Table 2 says", table2['Diffusion Policy'][0][1], "and", table2['GR00T-N1-2B'][0][1], "| Table 4 says 46.9 and 58.5")Output:
BC-Transformer plain mean of the 3 columns 32.1 | mean weighted by task count 26.4 | stated 26.4 Diffusion Policy plain mean of the 3 columns 38.1 | mean weighted by task count 33.4 | stated 33.4 GR00T-N1-2B plain mean of the 3 columns 49.5 | mean weighted by task count 45.1 | stated 45.0 -> the 'Average' column is the average over all 57 tasks, not over the 3 benchmarks (lesson 8.1: micro vs macro). GR-1 gap, GR00T minus Diffusion Policy: 17.3 points Diffusion Policy Table 4 / Figure 10 at 100 demos: [25.6, 46.9, 32.7] -> weighted average 32.0 GR00T-N1-2B Table 4 / Figure 10 at 100 demos: [32.1, 58.5, 50.0] -> weighted average 43.8 DexMG at 100 demos: Table 2 says 56.1 and 66.5 | Table 4 says 46.9 and 58.5
Run the example: Table 4, task by task
Our calculationWhat each part does
- Type in all 57 tasks × 6 numbers from Table 4 (Diffusion Policy and GR00T at 30, 100, 300 demonstrations).
- For each benchmark and size, count tasks where GR00T is ahead, level or behind, and average the columns.
- List the tasks where GR00T is behind at 300 demonstrations.
DATA = { "RoboCasa": [ ("Close Double Door", (1.7, 26.5, 60.8, 0.0, 43.1, 74.5)), ("Close Drawer", (57.5, 88.2, 94.1, 76.9, 96.1, 99.0)), ("Close Single Door", (21.7, 46.1, 72.6, 49.1, 67.7, 83.3)), ("Coffee Press Button", (32.5, 46.1, 91.2, 27.8, 56.9, 85.3)), ("Coffee Serve Mug", (6.7, 28.4, 66.7, 3.7, 34.3, 72.6)), ("Coffee Setup Mug", (0.0, 19.6, 32.4, 0.0, 2.0, 22.6)), ("Open Double Door", (0.0, 9.8, 18.6, 0.0, 12.8, 14.7)), ("Open Drawer", (15.8, 42.2, 61.8, 9.3, 42.2, 79.4)), ("Open Single Door", (36.7, 42.2, 57.8, 20.4, 54.9, 58.8)), ("PnP from Cab to Counter", (2.5, 4.9, 9.8, 0.9, 3.9, 19.6)), ("PnP from Counter to Cab", (0.0, 2.9, 10.8, 1.9, 6.9, 36.3)), ("PnP from Counter to Microwave", (0.0, 2.0, 8.8, 0.0, 0.0, 12.8)), ("PnP from Counter to Sink", (0.0, 0.0, 13.7, 0.0, 1.0, 9.8)), ("PnP from Counter to Stove", (0.0, 1.0, 17.7, 0.0, 0.0, 23.5)), ("PnP from Microwave to Counter", (0.0, 2.0, 11.8, 0.0, 0.0, 15.7)), ("PnP from Sink to Counter", (4.2, 8.8, 42.2, 0.0, 5.9, 33.3)), ("PnP from Stove to Counter", (1.7, 2.9, 23.5, 0.0, 0.0, 29.4)), ("Turn Off Microwave", (63.3, 53.9, 52.0, 47.2, 57.8, 70.6)), ("Turn Off Sink Faucet", (21.7, 63.7, 72.6, 49.1, 67.7, 72.6)), ("Turn Off Stove", (5.0, 10.8, 19.6, 4.6, 15.7, 26.5)), ("Turn On Microwave", (30.0, 51.0, 75.5, 55.6, 73.5, 78.4)), ("Turn On Sink Faucet", (31.7, 27.5, 63.7, 33.3, 59.8, 62.8)), ("Turn On Stove", (12.5, 22.6, 36.3, 14.8, 25.5, 55.9)), ("Turn Sink Spout", (8.3, 11.8, 23.5, 24.1, 42.2, 52.9)), ], "DexMG": [ ("Can Sort", (82.8, 93.1, 99.4, 94.8, 98.0, 98.0)), ("Coffee", (35.5, 68.1, 79.7, 44.9, 79.4, 73.5)), ("Pouring", (37.0, 62.3, 68.8, 54.4, 71.6, 87.3)), ("Threading", (4.2, 18.3, 27.5, 3.9, 37.3, 60.8)), ("Three Piece Assembly", (10.0, 32.5, 63.3, 10.8, 43.1, 69.6)), ("Transport", (7.5, 25.0, 53.3, 7.8, 48.0, 61.8)), ("Box Cleanup", (30.0, 80.8, 97.5, 33.3, 29.4, 95.1)), ("Drawer Cleanup", (1.7, 16.7, 52.5, 10.8, 42.2, 55.9)), ("Lift Tray", (5.0, 25.0, 73.3, 5.8, 77.5, 65.7)), ], "GR-1": [ ("Cutting Board to Pot", (22.6, 37.3, 48.0, 58.8, 57.8, 57.8)), ("Cutting Board to Basket", (19.6, 42.2, 29.4, 43.1, 61.8, 56.9)), ("Cutting Board to Tiered Basket", (13.7, 13.7, 18.6, 13.7, 23.5, 34.3)), ("Cutting Board to Pan", (28.4, 48.0, 57.8, 67.7, 65.7, 68.6)), ("Cutting Board to Cardboard Box", (11.8, 15.7, 22.6, 31.4, 30.4, 33.3)), ("Placemat to Bowl", (14.7, 18.6, 23.5, 31.4, 39.2, 39.2)), ("Placemat to Plate", (15.7, 23.5, 37.3, 33.3, 37.3, 49.0)), ("Placemat to Basket", (15.7, 25.5, 41.2, 50.0, 46.1, 55.9)), ("Placemat to Tiered Shelf", (6.9, 5.9, 11.8, 11.8, 21.6, 19.6)), ("Plate to Pan", (13.7, 17.7, 35.3, 35.3, 48.0, 52.9)), ("Plate to Cardboard Box", (12.8, 13.7, 27.5, 34.3, 38.2, 32.4)), ("Plate to Bowl", (15.7, 18.6, 31.4, 41.2, 42.2, 34.3)), ("Plate to Plate", (25.5, 39.2, 61.8, 72.6, 85.3, 68.6)), ("Tray to Tiered Shelf", (2.0, 6.9, 15.7, 17.7, 27.5, 14.7)), ("Tray to Tiered Basket", (12.8, 34.3, 39.2, 33.3, 49.0, 45.1)), ("Tray to Plate", (26.5, 41.2, 49.0, 53.9, 68.6, 62.8)), ("Tray to Cardboard Box", (21.6, 37.3, 40.2, 51.0, 55.9, 54.9)), ("Tray to Pot", (21.6, 48.0, 52.9, 52.0, 59.8, 65.7)), ("Wine to Cabinet", (43.1, 55.9, 60.8, 57.8, 53.9, 62.8)), ("Place Bottle to Cabinet", (40.2, 62.8, 60.8, 60.8, 81.4, 74.5)), ("Place Milk to Microwave", (37.3, 41.2, 51.0, 42.2, 58.8, 49.0)), ("Potato to Microwave", (17.7, 30.4, 41.2, 30.4, 26.5, 34.3)), ("Cup to Drawer", (24.5, 32.4, 36.3, 36.3, 44.1, 40.2)), ("Can to Drawer", (48.0, 74.5, 75.5, 77.5, 76.5, 75.5)), ], } sizes = [30, 100, 300] print("benchmark demos | DP avg | GR00T avg | GR00T ahead / level / behind (tasks)") for bench, rows in DATA.items(): for i, k in enumerate(sizes): dp = [v[i] for _, v in rows] gr = [v[3 + i] for _, v in rows] ahead = sum(g > d for g, d in zip(gr, dp)) behind = sum(g < d for g, d in zip(gr, dp)) level = len(rows) - ahead - behind print(f"{bench:<8} {k:>5} | {sum(dp) / len(rows):6.1f} | {sum(gr) / len(rows):9.1f} | {ahead:2d} / {level:2d} / {behind:2d}") print() print("GR-1 GR00T average, 100 -> 300 demos:", round(sum(v[4] for _, v in DATA['GR-1']) / 24, 1), "->", round(sum(v[5] for _, v in DATA['GR-1']) / 24, 1)) print("tasks where GR00T is behind Diffusion Policy at 300 demos:", [n for b in DATA.values() for n, v in b if v[5] < v[2]])Output:
benchmark demos | DP avg | GR00T avg | GR00T ahead / level / behind (tasks) RoboCasa 30 | 14.7 | 17.4 | 8 / 6 / 10 RoboCasa 100 | 25.6 | 32.1 | 16 / 1 / 7 RoboCasa 300 | 43.2 | 49.6 | 17 / 1 / 6 DexMG 30 | 23.7 | 29.6 | 8 / 0 / 1 DexMG 100 | 46.9 | 58.5 | 8 / 0 / 1 DexMG 300 | 68.4 | 74.2 | 5 / 0 / 4 GR-1 30 | 21.3 | 43.2 | 23 / 1 / 0 GR-1 100 | 32.7 | 50.0 | 22 / 0 / 2 GR-1 300 | 40.4 | 49.3 | 20 / 1 / 3 GR-1 GR00T average, 100 -> 300 demos: 50.0 -> 49.3 tasks where GR00T is behind Diffusion Policy at 300 demos: ['Coffee Press Button', 'Coffee Setup Mug', 'Open Double Door', 'PnP from Counter to Sink', 'PnP from Sink to Counter', 'Turn On Sink Faucet', 'Can Sort', 'Coffee', 'Box Cleanup', 'Lift Tray', 'Tray to Tiered Shelf', 'Place Milk to Microwave', 'Potato to Microwave']
- Evidence in the paper
- §4.4 “Simulation” · Table 2 · Table 4 (Appendix B) · Figure 10
Check your understanding: GR00T has the higher average in all nine benchmark-and-size cases. Does it win every task?
No. For example with 30 RoboCasa demonstrations it is ahead on 8 tasks, level on 6 and behind on 10. The claim holds for averages, not for every task.
Real-robot results and data efficiency
- Intuition
- The real test: a physical humanoid, real objects and a small amount of teaching. “Data efficiency” asks how much you lose if you give it one tenth of the demonstrations.
- How it works
- GR00T-N1-2B is compared with Diffusion Policy on 13 tasks, trained with 10% and with all of the task data.
- Technical details
Table 3, regrouped by us so that equal data sit side by side. Task-weighted overall; partial credit counts. Success rate (%) Diffusion Policy, 10% data GR00T, 10% data Diffusion Policy, all data GR00T, all data Pick-and-place (5 tasks) 3.0 35.0 36.0 82.0 Articulated (3) 14.3 62.0 38.6 70.9 Industrial (3) 6.7 31.0 61.0 70.0 Coordination (2) 27.5 50.0 62.5 82.5 Overall (13 tasks) 10.2 42.6 46.4 76.8 GR00T leads by about 32 points with 10% of the data and about 30 points with all of it (percentage points, not relative gains). With 10% of the data it scores 42.6, close to Diffusion Policy’s 46.4 with all of it. The authors read this as data efficiency. Our interpretation With about ten trials per task the uncertainty on each average is roughly ±8 points, so “about as good as the baseline with ten times less data” is a fair reading, but “3.8 points lower” is not a meaningful difference. The groups also differ: in the industrial tasks GR00T at 10% is far below Diffusion Policy at full data (31.0 against 61.0), in articulated tasks far above (62.0 against 38.6).
Run the example: check Tables 3 and 5, with uncertainty
Our calculationWhat each part does
- Type in the per-task scores from Table 5 and rebuild each category average and the overall one.
- Compare with Table 3 and compute the gaps between GR00T and Diffusion Policy.
- Assume 10 trials per task to put 95% intervals (Wilson) on each overall score, and list tasks where less data scored higher than more.
import math # Table 5, per task: [Diffusion Policy 10%, Diffusion Policy full, GR00T 10%, GR00T full] pnp_seen = [[0.0, 20.0, 40.0, 100.0], [0.0, 30.0, 10.0, 100.0], [0.0, 60.0, 60.0, 80.0], [0.0, 40.0, 30.0, 100.0], [10.0, 60.0, 40.0, 80.0]] pnp_unseen = [[0.0, 20.0, 30.0, 80.0], [10.0, 20.0, 60.0, 60.0], [0.0, 40.0, 40.0, 80.0], [0.0, 20.0, 10.0, 40.0], [10.0, 50.0, 30.0, 100.0]] articulated = {"White Drawer": [6.6, 36.4, 26.4, 79.9], "Dark Cabinet": [0.0, 46.2, 86.6, 69.7], "Wooden Chest": [36.4, 33.2, 72.9, 63.2]} industrial = {"Machinery Packing": [20.0, 44.0, 8.0, 56.0], "Mesh Cup Pouring": [0.0, 62.5, 65.0, 67.5], "Cylinder Handover": [0.0, 76.5, 20.0, 86.6]} coordination = {"Part 1": [45.0, 65.0, 70.0, 80.0], "Part 2": [10.0, 60.0, 30.0, 85.0]} def mean_col(rows, j): return sum(r[j] for r in rows) / len(rows) names = ["Diffusion Policy 10%", "Diffusion Policy full", "GR00T 10%", "GR00T full"] cats = { "Pick-and-Place (5 tasks)": [[(a[j] + b[j]) / 2 for j in range(4)] for a, b in zip(pnp_seen, pnp_unseen)], # each task is run with seen and unseen objects "Articulated (3)": list(articulated.values()), "Industrial (3)": list(industrial.values()), "Coordination (2)": list(coordination.values()), } print("category averages, recomputed (Table 3 prints 3.0 / 14.3 / 6.7 / 27.5 for Diffusion Policy 10%):") overall = [0, 0, 0, 0]; total_tasks = 0 for cat, rows in cats.items(): avg = [mean_col(rows, j) for j in range(4)] print(f" {cat:<26}", [round(a, 1) for a in avg]) for j in range(4): overall[j] += avg[j] * len(rows) total_tasks += len(rows) overall = [o / total_tasks for o in overall] print("overall over 13 tasks:", [round(o, 1) for o in overall], "(Table 3 prints 10.2, 46.4, 42.6, 76.8)") print("GR00T minus Diffusion Policy at 10% data:", round(overall[2] - overall[0], 1), "| with all data:", round(overall[3] - overall[1], 1), "| GR00T 10% vs Diffusion Policy full:", round(overall[2] - overall[1], 1)) def wilson(p, n, z=1.96): centre = (p + z * z / (2 * n)) / (1 + z * z / n) half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n) return centre - half, centre + half print() print("if each of the 13 tasks had 10 trials (the paper says 5 for one task, and uses partial scores), n is about 130:") for name, o in zip(names, overall): lo, hi = wilson(o / 100, 130) print(f" {name:<22} {o:5.1f}% 95% interval {100 * lo:4.1f}% to {100 * hi:4.1f}%") print() print("per-task cases where LESS data scored higher than MORE data (10% > full):") for group in (articulated, industrial, coordination): for task, v in group.items(): if v[0] > v[1]: print(" Diffusion Policy,", task, ":", v[0], "(10%) vs", v[1], "(full)") if v[2] > v[3]: print(" GR00T,", task, ":", v[2], "(10%) vs", v[3], "(full)")Output:
category averages, recomputed (Table 3 prints 3.0 / 14.3 / 6.7 / 27.5 for Diffusion Policy 10%): Pick-and-Place (5 tasks) [3.0, 36.0, 35.0, 82.0] Articulated (3) [14.3, 38.6, 62.0, 70.9] Industrial (3) [6.7, 61.0, 31.0, 70.0] Coordination (2) [27.5, 62.5, 50.0, 82.5] overall over 13 tasks: [10.2, 46.4, 42.6, 76.8] (Table 3 prints 10.2, 46.4, 42.6, 76.8) GR00T minus Diffusion Policy at 10% data: 32.4 | with all data: 30.3 | GR00T 10% vs Diffusion Policy full: -3.8 if each of the 13 tasks had 10 trials (the paper says 5 for one task, and uses partial scores), n is about 130: Diffusion Policy 10% 10.2% 95% interval 6.1% to 16.6% Diffusion Policy full 46.4% 95% interval 38.1% to 55.0% GR00T 10% 42.6% 95% interval 34.4% to 51.2% GR00T full 76.8% 95% interval 68.8% to 83.2% per-task cases where LESS data scored higher than MORE data (10% > full): GR00T, Dark Cabinet : 86.6 (10%) vs 69.7 (full) Diffusion Policy, Wooden Chest : 36.4 (10%) vs 33.2 (full) GR00T, Wooden Chest : 72.9 (10%) vs 63.2 (full)
Run the example: how certain is a result from 15 trials?
Our calculationWhat each part does
- Define the Wilson interval, a standard way to put a 95% range on a success rate measured over a few trials.
- Apply it to the two pre-trained-model results (11.5 and 11 successes out of 15).
import math def wilson(successes, n, z=1.96): # 95% interval for a success rate (lesson 8.2) p = successes / n centre = (p + z * z / (2 * n)) / (1 + z * z / n) half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n) return centre - half, centre + half for name, score in (("handover task, 11.5 of 15", 11.5), ("novel object task, 11 of 15", 11.0)): lo, hi = wilson(score, 15) print(f"{name}: {100 * score / 15:.1f}% -> 95% interval {100 * lo:.0f}% to {100 * hi:.0f}%") print("(half-credit trials make this approximate)")Output:
handover task, 11.5 of 15: 76.7% -> 95% interval 51% to 91% novel object task, 11 of 15: 73.3% -> 95% interval 48% to 89% (half-credit trials make this approximate)
Half-credit trials make this approximate; it is meant to show the size of the uncertainty, not an exact range.
- Evidence in the paper
- §4.4 “Real-world” · Table 3 · Table 5 (Appendix B)
Check your understanding: GR00T at 10% data scored 42.6 and Diffusion Policy at full data 46.4. Is Diffusion Policy better?
Not demonstrably. With roughly 130 trials, the 95% intervals overlap widely. What the data supports is “comparable”, with GR00T using a tenth of the demonstrations.
Does generated video help? (neural trajectories)
- Intuition
- If a robot has only a few real lessons, do extra “imagined” lessons help?
- How it works
- In RoboCasa, 3,000 generated trajectories per task, labelled either with latent actions or with the IDM, are added during post-training. On the real robot, 100 generated trajectories per task with IDM labels are added to the 10% regime for eight tasks.
- Technical details
Figure 9 numbers read from the figure labels. RoboCasa success (%) 30 demos 100 demos 300 demos Diffusion Policy 14.7 25.6 43.2 GR00T 17.4 32.1 49.6 GR00T + latent-action labels 21.6 39.5 52.9 GR00T + IDM labels 21.2 40.9 56.4 The paper reports gains of +4.2, +8.8 and +6.8 points for 30, 100 and 300 demonstrations. Each is the better of the two label types in that regime, so the typical gain is a little smaller. Our calculation On the real robot (10% data, eight tasks) the average rises from 33.8 to 39.6, about +5.8. The authors call these “preliminary results”.
Run the example: the Figure 9 gains
Our calculationWhat each part does
- Type in the bar labels for the three RoboCasa regimes and the real-robot averages.
- Compute the gain from each label type over GR00T alone.
- Check that the paper’s +4.2, +8.8 and +6.8 are the larger of the two gains.
# RoboCasa average success (%), read from Figure 9: demonstrations per task -> (Diffusion Policy, GR00T, GR00T + LAPA labels, GR00T + IDM labels) robocasa = {30: (14.7, 17.4, 21.6, 21.2), 100: (25.6, 32.1, 39.5, 40.9), 300: (43.2, 49.6, 52.9, 56.4)} print("demos | gain with LAPA | gain with IDM | the larger of the two") for demos, (dp, g, lapa, idm) in robocasa.items(): print(f"{demos:>5} | {lapa - g:+13.1f} | {idm - g:+13.1f} | {max(lapa, idm) - g:+.1f}") print("the paper's +4.2, +8.8, +6.8 are the larger of the two in each regime.") print() print("LAPA minus IDM:", {d: round(v[2] - v[3], 1) for d, v in robocasa.items()}) real = {"Pick-and-Place (seen)": (2.0, 36.0, 42.0), "Industrial": (6.67, 31.0, 36.67), "Overall (8 tasks)": (4.07, 33.78, 39.63)} print() for name, (dp, g, idm) in real.items(): print(f"real GR-1, 10% data, {name}: Diffusion Policy {dp}, GR00T {g}, GR00T + neural (IDM) {idm} -> gain {idm - g:+.2f}")Output:
demos | gain with LAPA | gain with IDM | the larger of the two 30 | +4.2 | +3.8 | +4.2 100 | +7.4 | +8.8 | +8.8 300 | +3.3 | +6.8 | +6.8 the paper's +4.2, +8.8, +6.8 are the larger of the two in each regime. LAPA minus IDM: {30: 0.4, 100: -1.4, 300: -3.5} real GR-1, 10% data, Pick-and-Place (seen): Diffusion Policy 2.0, GR00T 36.0, GR00T + neural (IDM) 42.0 -> gain +6.00 real GR-1, 10% data, Industrial: Diffusion Policy 6.67, GR00T 31.0, GR00T + neural (IDM) 36.67 -> gain +5.67 real GR-1, 10% data, Overall (8 tasks): Diffusion Policy 4.07, GR00T 33.78, GR00T + neural (IDM) 39.63 -> gain +5.85- Evidence in the paper
- §4.4 “Neural trajectories” · Figure 9
Check your understanding: Why might the paper’s headline gains slightly overstate the typical benefit?
Each reported gain picks the better of two labelling methods in its regime, which is a selection among two outcomes.
What the results show
- With the same fine-tuning data, the pre-trained GR00T model scores higher on average than the two from-scratch baselines on all three simulated benchmarks and on the real humanoid tasks.
- The gap is largest where data is scarce (GR-1 benchmark, 10% real-robot data) and is far larger than the noise in the real-robot averages.
- Adding generated video helped in the reported tests.
- The pre-trained model completed two real tasks at 73 to 77% (15 trials each).
What they do not show
- Which ingredient matters: the architecture, the vision-language backbone, the data pyramid or sheer scale. There are no ablations of those choices.
- That GR00T wins every task or that more fine-tuning data always helps (see the per-task and plateau notes in Part F).
- That simulated scores predict real-robot results.
- That generated data is accurate, or that the gains survive repeated runs: no repeated runs or confidence intervals are reported.
- How the model performs on long tasks, which the authors list as a limitation.
Part E · optional
Engineering deep dive
Equations, tensor shapes, hyperparameters and implementation details, for readers who want to rebuild or audit the model. Everything here is collapsed; Parts A to D do not depend on it.
Flow matching: the equations
Symbols: At is the true chunk of 16 actions starting at time t; ε is random noise; τ (tau) is a number between 0 and 1 that says how far we are from noise to data; φt are the vision-language features; qt is the robot state; Vθ is the action network.
Reported by the paper A noisy chunk is a blend of data and noise (τ = 0 is pure noise, τ = 1 is the clean chunk):
Reported by the paper Training loss, Equation 1 as printed. The network is asked to predict ε − At:
Reported by the paper The training time is not uniform. It is drawn so that (s − τ)/s follows a Beta(1.5, 1) distribution with s = 0.999 (as in π0), which places more training effort on the noisy end (small τ):
Reported by the paper At run time, start from noise and take K = 4 small steps:
Our calculation A sign question. With this blend, the direction in which the chunk moves as τ grows is At − ε, the opposite of the printed target ε − At, while the update adds +V/K. The examples below test it. See finding F1. For the background, see lesson 4.7.
Run the example: Equation 1 in code and the sign of its target
What each part does
- Draw a random chunk A and noise ε.
- Implement the blend and check τ = 0 gives noise and τ = 1 gives the data.
- Measure how the blend moves with τ numerically and compare with A − ε and ε − A.
- Compute the loss for a perfect predictor of the printed target and for one that predicts A − ε.
import numpy as np
rng = np.random.default_rng(0)
H, action_dim = 16, 7
A = rng.normal(size=(H, action_dim)) # a clean action chunk (random numbers stand in for real actions)
eps = rng.normal(size=(H, action_dim)) # sampled noise
def noised(A, eps, tau): # the paper's blend: tau = 0 is noise, tau = 1 is the clean chunk
return tau * A + (1 - tau) * eps
print("tau = 0 gives pure noise:", bool(np.allclose(noised(A, eps, 0.0), eps)))
print("tau = 1 gives the clean chunk:", bool(np.allclose(noised(A, eps, 1.0), A)))
d = 1e-6
slope = (noised(A, eps, 0.5 + d) - noised(A, eps, 0.5)) / d # how the blend moves as tau increases
print("the blend moves in the direction A - eps as tau grows:", bool(np.allclose(slope, A - eps, atol=1e-4)))
print("the paper's printed regression target is eps - A, the opposite direction:", bool(np.allclose(eps - A, -slope, atol=1e-4)))
def loss(prediction): # Equation 1 for one sample: squared error against the printed target
return float(np.mean((prediction - (eps - A)) ** 2))
print("loss of a perfect predictor of the printed target:", loss(eps - A))
print("loss of a predictor that moves along A - eps instead:", round(loss(A - eps), 3))Output:
tau = 0 gives pure noise: True tau = 1 gives the clean chunk: True the blend moves in the direction A - eps as tau grows: True the paper's printed regression target is eps - A, the opposite direction: True loss of a perfect predictor of the printed target: 0.0 loss of a predictor that moves along A - eps instead: 8.507
Run the example: which noise levels does training visit?
What each part does
- Sample u from Beta(1.5, 1) and convert to τ = s(1 − u).
- Print the mean, the share below 0.5 and a five-bin histogram, with the exact values for comparison.
import numpy as np
rng = np.random.default_rng(0)
s = 0.999
u = rng.beta(1.5, 1.0, 200000) # u = (s - tau) / s follows Beta(1.5, 1)
tau = s * (1 - u)
print("average tau:", round(float(tau.mean()), 3), "(exact value:", round(s * (1 - 1.5 / 2.5), 3), ")")
print("share of training samples with tau below 0.5:", round(float(np.mean(tau < 0.5)), 3), "(exact:", round(1 - 0.5 ** 1.5, 3), ")")
edges = np.linspace(0, 1, 6)
counts, _ = np.histogram(tau, bins=edges)
for lo, hi, n in zip(edges[:-1], edges[1:], counts):
print(f"tau in [{lo:.1f}, {hi:.1f}): {n / len(tau):.3f}")Output:
average tau: 0.399 (exact value: 0.4 ) share of training samples with tau below 0.5: 0.648 (exact: 0.646 ) tau in [0.0, 0.2): 0.286 tau in [0.2, 0.4): 0.250 tau in [0.4, 0.6): 0.211 tau in [0.6, 0.8): 0.164 tau in [0.8, 1.0): 0.089
Run the example: do the four steps carry noise to data, with each sign?
What each part does
- Use toy data (a single action equal to −1 or +1) for which the exact velocity field has a formula, and check it against a brute-force average.
- Run the K-step update with V = A − ε and with V = ε − A and count how many samples land near a data point.
- Repeat for K = 1, 2, 4 and 8.
import numpy as np
rng = np.random.default_rng(0)
# Toy data: a single action that is either -1 or +1. For this data the exact velocity field has a closed form:
# v(x, tau) = E[A - eps | x] = (tanh(tau * x / (1 - tau)**2) - x) / (1 - tau)
def v(x, tau):
return (np.tanh(tau * x / (1 - tau) ** 2) - x) / (1 - tau)
# check the formula against a brute-force conditional average at one point
tau0, x0 = 0.5, 0.3
A = rng.choice([-1.0, 1.0], 4_000_000); e = rng.normal(size=A.size)
x = tau0 * A + (1 - tau0) * e
near = np.abs(x - x0) < 0.01
print("formula:", round(float(v(x0, tau0)), 2), "| brute force:", round(float((A[near] - e[near]).mean()), 2))
def sample(sign, K=4, n=20000):
x = rng.normal(size=n) # start from noise at tau = 0
for k in range(K):
x = x + sign * v(x, k / K) / K # the paper's update: A <- A + (1/K) * V
return x
for sign, name in ((+1, "V = A - eps (the true derivative)"), (-1, "V = eps - A (the printed target)")):
out = sample(sign)
near_data = np.minimum(np.abs(out - 1), np.abs(out + 1)) < 0.5
print(f"{name}: {100 * near_data.mean():5.1f}% of samples end within 0.5 of a data point")
print()
for K in (1, 2, 4, 8):
out = sample(+1, K)
print(f"K = {K}: {100 * np.mean(np.minimum(np.abs(out - 1), np.abs(out + 1)) < 0.1):5.1f}% end within 0.1 of a data point")Output:
formula: 0.47 | brute force: 0.46 V = A - eps (the true derivative): 96.1% of samples end within 0.5 of a data point V = eps - A (the printed target): 28.6% of samples end within 0.5 of a data point K = 1: 0.0% end within 0.1 of a data point K = 2: 13.9% end within 0.1 of a data point K = 4: 89.1% end within 0.1 of a data point K = 8: 99.8% end within 0.1 of a data point
A toy world, not NVIDIA’s model. It tests the mathematics of the update rule, nothing about GR00T’s accuracy.
Inspect tensor shapes
| Stage | Shape | Source |
|---|---|---|
| Camera image | 224 × 224 × 3 | Reported (§2.1) |
| After pixel shuffle | 64 tokens per frame; token width not stated | Reported tokens; width not given |
| Vision-language features used by System 1 | batch × sequence length × hidden size, from the 12th layer | Reported (§2.1) |
| Robot state | one vector, size depends on the robot | Reported; our toy uses sizes 4, 7 and 29 |
| Noisy action chunk | 16 × (action size of this robot) | Reported (H = 16) |
| Inside the DiT (toy) | state token + 16 action tokens, width 8 → 17 × 8 | Our toy; the real width is not stated |
| Output after the per-robot decoder | 16 × (action size of this robot) | Reported (§2.1) |
Hyperparameters (Table 6)
| Setting | Pre-training | Post-training |
|---|---|---|
| Learning rate / optimiser | 1e-4, AdamW (β₁ 0.95, β₂ 0.999, ε 1e-8) | same |
| Weight decay / schedule | 1e-5, cosine, warm-up ratio 0.05 | same |
| Batch size | 16,384 | 128 or 1,024 |
| Gradient steps | 200,000 | 20,000 to 60,000 |
| Vision encoder / text tokenizer / DiT | unfrozen / frozen / unfrozen | same |
| Cluster | up to 1,024 H100 GPUs, about 50,000 GPU-hours | single A6000 tested: batch up to 200 (adapters and DiT) or 16 (with the vision encoder) |
Run the example: the compute and batch arithmetic
What each part does
- Divide 50,000 GPU-hours by 1,024 GPUs to get a wall-clock estimate.
- Multiply batch size by steps to get the number of samples seen.
- Divide the batch among the GPUs and estimate GPU-seconds per sample.
gpu_hours, gpus = 50_000, 1024
print("pre-training wall-clock if all 1,024 GPUs ran the whole time:", round(gpu_hours / gpus, 1), "hours =", round(gpu_hours / gpus / 24, 1), "days")
batch, steps = 16_384, 200_000 # pre-training, Table 6
print("samples seen in pre-training:", f"{batch * steps:,}", "=", round(batch * steps / 1e9, 2), "billion")
print("per-GPU batch on 1,024 GPUs:", batch // gpus)
print("GPU-seconds per sample:", round(gpu_hours * 3600 / (batch * steps), 3))Output:
pre-training wall-clock if all 1,024 GPUs ran the whole time: 48.8 hours = 2.0 days samples seen in pre-training: 3,276,800,000 = 3.28 billion per-GPU batch on 1,024 GPUs: 16 GPU-seconds per sample: 0.055
Data format and how action and state spaces are unified (Appendix E)
Reported by the paper Training data uses Hugging Face’s LeRobot format (parquet tables, MP4 video, JSON metadata), extended with: a modality.json file mapping each state and action dimension to a meaning; named fields (end-effector position, orientation, gripper) with their own ranges and transforms; several annotation types per dataset; and an explicit rotation representation. It stays backward compatible with LeRobot.
Reported by the paper Spaces are unified on a best-effort basis: end-effector rotation states use a 6D representation (avoids Euler-angle discontinuities); rotation actions use axis-angle; joint and end-effector values are min-max normalised; vectors follow a fixed order (rotation, position, gripper; left arm then right arm). See lesson 4.3 and lesson 4.1.
Run the example: why min-max and 6D rotations
What each part does
- Rescale a gripper opening from two robots with different units to the same range.
- Compare two nearly identical yaw angles that sit on opposite sides of the ±180° jump in Euler angles, with and without a 6D representation.
import math
def minmax(x, lo, hi):
return 2 * (x - lo) / (hi - lo) - 1 # rescale to [-1, 1]
# two robots whose gripper opening is measured in different units
robot_a = [0.0, 0.04, 0.08] # metres, range 0 .. 0.08
robot_b = [0, 50, 100] # percent, range 0 .. 100
print("robot A:", [round(minmax(v, 0.0, 0.08), 2) for v in robot_a])
print("robot B:", [round(minmax(v, 0, 100), 2) for v in robot_b])
# Euler angles are discontinuous at +/-180 degrees; a 6D rotation (two columns of the matrix) is not
def rot_z(deg):
t = math.radians(deg)
return [[math.cos(t), -math.sin(t), 0], [math.sin(t), math.cos(t), 0], [0, 0, 1]]
def six_d(m):
return [m[0][0], m[1][0], m[2][0], m[0][1], m[1][1], m[2][1]]
for a, b in [(179, -179)]:
near = math.dist(six_d(rot_z(a)), six_d(rot_z(b)))
print(f"yaw {a} vs {b} degrees: Euler numbers differ by {abs(a - b)} but the 6D vectors differ by only {near:.3f}")Output:
robot A: [-1.0, 0.0, 1.0] robot B: [-1.0, 0.0, 1.0] yaw 179 vs -179 degrees: Euler numbers differ by 358 but the 6D vectors differ by only 0.049
The auxiliary object-detection loss (Appendix F)
Reported by the paper To improve spatial understanding, the model also predicts where the target object is. For each frame, an off-the-shelf detector (OWL-v2) gives a box around the object named in the instruction. The box centre, divided by the image width and height, is the label. A linear layer on top of the final vision-language tokens predicts it, with a squared-error loss, and the total loss is the flow-matching loss plus this one. The paper reports no experiment with and without it, so its benefit is not shown.
Video-generation and IDM pipeline (Appendix F)
- Reported by the paper Video model: Wan 2.1 image-to-video (14B), fine-tuned with LoRA (training small added matrices) on teleoperation trajectories downsampled to 81 frames at 480P.
- Reported by the paper Quality filter: eight frames per video are shown to a commercial multimodal model that judges whether the video follows its instruction. Failures are re-captioned (16 frames at 256P) rather than discarded.
- Reported by the paper IDM: conditioned on a current and a future frame, outputs the action chunk between them; same DiT as System 1 with SigLIP-2 embeddings and flow matching; trained per embodiment for 30K or 60K steps; used to pseudo-label every step of the generated videos.
Datasets used for pre-training (Table 7, totals)
| Group | Frames | Hours |
|---|---|---|
| Real robot (GR-1 teleoperation, seven Open X-Embodiment datasets, AgiBot-Alpha) | 262.3 M | 3,288.8 |
| Human video (seven datasets) | 181.3 M | 2,517.0 |
| Simulation (DexMimicGen) | 125.5 M | 1,742.6 |
| Generated robot video (neural trajectories) | 23.8 M | 827.3 |
| All | 592.9 M | 8,375.7 |
Our interpretation Web data used to train the vision-language model is not part of this table.
Part F
Critical paper review
Everything above reports what the paper says. This part is our independent reading: places where the paper’s own numbers disagree, where wording is ambiguous, where the method limits what can be concluded, and what the authors themselves list as limits. We looked for these in good faith; none of them overturns the paper’s main message, and each is linked back to the concept it affects. Calculations are reproducible in the examples.
Numerical inconsistencies and ambiguities
-
F1. The sign of the flow-matching target Numerical inconsistency
What the paper says. Noisy chunks are τ·A + (1 − τ)·ε (τ = 0 is noise). Equation 1 trains the network to predict ε − A, and sampling adds V/K at each step.
What we found. With that blend the chunk moves in the direction A − ε as τ grows. Taking steps along the printed target moves away from the data. In our toy test the update with the printed sign lands near a data point for 28.6% of samples, against 96.1% with A − ε.
Why it matters. Probably a typo or a convention difference (the authors’ code is the authority). It matters if you re-implement the equations literally; it does not by itself change the reported results. Concept: System 1 · example
-
F2. Two different DexMG numbers for the same setting Numerical inconsistency
What the paper says. Table 2 (100 demonstrations): DexMG 56.1 for Diffusion Policy and 66.5 for GR00T. Table 4 and Figure 10 for the same 100-demonstration setting give 46.9 and 58.5.
What we found. The RoboCasa and GR-1 columns match between the tables; only DexMG differs. Using Table 4’s DexMG, the weighted averages would be 32.0 and 43.8 instead of 33.4 and 45.0. Our recomputation otherwise reproduces the printed Table 2 averages to within rounding, including that “Average” is task-weighted over 57 tasks.
Why it matters. The gap between the two models is about the same either way (11.6 to 11.8 points), so the conclusion holds, but one of the two tables has a wrong or differently defined number. Concept: Simulation results · example
-
F3. The 780,000 simulated trajectories cannot be itemised Ambiguity
What the paper says. 780,000 simulated trajectories (equal to 6,500 hours) were generated across pre- and post-training. Pre-training uses 54 container pairs × 10,000 = 540,000.
What we found. From the numbers printed in the text we can account for about 573,000 (540,000 plus 9 × 1,000 for the DexMG suite plus 24 × 1,000 for GR-1 Tabletop). About 207,000 are not itemised; RoboCasa’s own 3,000 demonstrations per task are generated by MimicGen and not counted here.
Why it matters. A headline figure in the abstract-level story (“780,000 trajectories”) is not fully traceable. It probably includes other datasets the paper does not list; we cannot tell. Concept: Simulation data
Run the example: itemising the simulation data
Our calculationWhat each part does
- Multiply the 54 container pairs by 10,000.
- Add the DexMG and GR-1 Tabletop benchmark datasets.
- Compare with 780,000.
pairs, per_pair = 54, 10_000 pretraining_sim = pairs * per_pair print("pre-training simulation demonstrations:", f"{pretraining_sim:,}", "(paper: 540k)") # post-training simulation sets described in Section 4.1: demonstrations generated with DexMimicGen dexmg = 9 * 1000 # DexMG suite: 9 tasks x 1000 gr1 = 24 * 1000 # GR-1 tabletop: 24 tasks x 1000 itemised = pretraining_sim + dexmg + gr1 print("itemised so far (pre-training + DexMG + GR-1 tabletop):", f"{itemised:,}") print("the paper says 780,000 in total ->", f"{780_000 - itemised:,}", "demonstrations are not itemised in the text") print("(RoboCasa's 3,000 demos per task were generated by MimicGen for the RoboCasa benchmark, so they are not counted above)")Output:
pre-training simulation demonstrations: 540,000 (paper: 540k) itemised so far (pre-training + DexMG + GR-1 tabletop): 573,000 the paper says 780,000 in total -> 207,000 demonstrations are not itemised in the text (RoboCasa's 3,000 demos per task were generated by MimicGen for the RoboCasa benchmark, so they are not counted above)
-
F4. Table 7: Ego-Exo4D hours do not match its frames Numerical inconsistency
What the paper says. Table 7 lists frames, frame rate and hours for every dataset, with totals of 592.9 M frames and 8,375.7 hours.
What we found. The rows add up to the printed totals exactly. Frames ÷ frame rate reproduces the stated hours for every row except Ego-Exo4D: 8.9 M frames at 30 FPS is about 82 hours, but the table says 123 hours (perhaps more camera views are counted; the paper does not say). The table also shows more real-robot hours (about 3,289) than human-video hours (about 2,517), which a reader of the pyramid picture would not expect.
Why it matters. A small bookkeeping question (about 40 hours out of 8,376). The pyramid figure should be read as a picture of data types, not of hours. Concept: Data pyramid
Run the example: Table 7 re-added
Our calculationWhat each part does
- Type in each dataset row (frames, frames per second, stated hours).
- Add the rows by group and overall, and compare with the printed totals.
- Recompute hours from frames and frame rate and list the rows that differ by more than 5%.
# (name, frames in millions, hours, frames per second, category) copied from Table 7 table7 = [ ("GR-1 teleoperation", 6.4, 88.4, 20, "robot"), ("DROID (OXE)", 23.1, 428.3, 15, "robot"), ("RT-1 (OXE)", 3.7, 338.4, 3, "robot"), ("Language Table (OXE)", 7.0, 195.7, 10, "robot"), ("Bridge-v2 (OXE)", 2.0, 111.1, 5, "robot"), ("MUTEX (OXE)", 0.362, 5.0, 20, "robot"), ("Plex (OXE)", 0.077, 1.1, 20, "robot"), ("RoboSet (OXE)", 1.4, 78.9, 5, "robot"), ("Agibot-Alpha", 213.8, 1979.4, 30, "robot"), ("RH20T-Robot", 4.5, 62.5, 20, "robot"), ("Ego4D", 154.4, 2144.7, 20, "human"), ("Ego-Exo4D", 8.9, 123.0, 30, "human"), ("Assembly-101", 1.4, 19.3, 20, "human"), ("HOI4D", 0.892, 12.4, 20, "human"), ("HoloAssist", 12.2, 169.6, 20, "human"), ("RH20T-Human", 1.2, 16.3, 20, "human"), ("EPIC-KITCHENS", 2.3, 31.7, 20, "human"), ("GR-1 simulation", 125.5, 1742.6, 20, "simulation"), ("GR-1 neural videos", 23.8, 827.3, 8, "neural"), ] stated = {"robot": (262.3, 3288.8), "human": (181.3, 2517.0), "simulation": (125.5, 1742.6), "neural": (23.8, 827.3)} for cat, (frames_total, hours_total) in stated.items(): frames = sum(r[1] for r in table7 if r[4] == cat); hours = sum(r[2] for r in table7 if r[4] == cat) print(f"{cat:<11} rows add to {frames:6.1f} M frames, {hours:7.1f} h | stated totals {frames_total} M, {hours_total} h") print("all data:", round(sum(r[1] for r in table7), 1), "M frames,", round(sum(r[2] for r in table7), 1), "h (stated 592.9 M, 8,375.7 h)") print() print("hours implied by frames / fps, compared with the stated hours (rows that differ by more than 5%):") for name, frames, hours, fps, cat in table7: implied = frames * 1e6 / fps / 3600 if abs(hours - implied) / implied > 0.05: print(f" {name}: stated {hours} h, frames/fps gives {implied:.1f} h") print() for cat, (_, hours_total) in stated.items(): print(f"share of the 8,375.7 hours: {cat:<11} {100 * hours_total / 8375.7:4.1f}%") print("Agibot-Alpha share of the robot hours:", round(100 * 1979.4 / 3288.8), "%") print("GR-1 teleoperation share of the robot hours:", round(100 * 88.4 / 3288.8, 1), "%")Output:
robot rows add to 262.3 M frames, 3288.8 h | stated totals 262.3 M, 3288.8 h human rows add to 181.3 M frames, 2517.0 h | stated totals 181.3 M, 2517.0 h simulation rows add to 125.5 M frames, 1742.6 h | stated totals 125.5 M, 1742.6 h neural rows add to 23.8 M frames, 827.3 h | stated totals 23.8 M, 827.3 h all data: 592.9 M frames, 8375.7 h (stated 592.9 M, 8,375.7 h) hours implied by frames / fps, compared with the stated hours (rows that differ by more than 5%): Ego-Exo4D: stated 123.0 h, frames/fps gives 82.4 h share of the 8,375.7 hours: robot 39.3% share of the 8,375.7 hours: human 30.1% share of the 8,375.7 hours: simulation 20.8% share of the 8,375.7 hours: neural 9.9% Agibot-Alpha share of the robot hours: 60 % GR-1 teleoperation share of the robot hours: 2.7 %
-
F5. Rates, chunk length and what “63.9 ms” covers Ambiguity
What the paper says. System 1 outputs actions at 120 Hz; a chunk has 16 actions and takes 63.9 ms to sample on an L40; teleoperation data was recorded at 20 Hz.
What we found. These can all be true together, but the paper does not say whether 63.9 ms includes the vision-language pass, how chunks are overlapped to give a steady 120 Hz stream, or what a 16-step chunk means in seconds (0.8 s at the 20 Hz data rate, 133 ms at 120 Hz). If 63.9 ms is the whole cost per chunk, a 120 Hz stream would use about half the GPU’s time.
Why it matters. Real-time claims need the exact schedule to be reproduced or trusted. Concept: Action chunks · example
-
F6. Small rounding and cost differences Numerical inconsistency
What the paper says. 76.6% for 11.5 of 15 trials; GR00T ahead by 30.4 points with all data; about 105,000 L40 GPU-hours, “roughly 1.5 days” on 3,600 GPUs.
What we found. 11.5 of 15 is 76.7%. 76.8 − 46.4 is 30.4, but unrounded values give 30.3. The cost estimate of 2 minutes of GPU time per second of video for 827 hours gives about 99,000 GPU-hours (6% below 105,000), and 105,000 GPU-hours on 3,600 GPUs is about 1.2 days, not 1.5.
Why it matters. Immaterial to conclusions; listed for completeness. The extra days may include overhead the paper does not describe. Concept: Neural trajectories · example
Methodological caveats
-
F7. Baselines are trained from scratch Method caveat
What the paper says. GR00T N1 outperforms “state-of-the-art imitation learning baselines”: BC-Transformer and Diffusion Policy.
What we found. Both baselines are trained only on the task data, with no pre-trained vision-language model and no cross-robot pre-training. GR00T differs in architecture and in pre-training at once.
Why it matters. The results support “the full GR00T recipe beats from-scratch policies”; they cannot say whether the architecture, the pre-trained backbone or the data is responsible. A fairer test would add a baseline with the same pre-training or the same backbone. Concept: The set-up
-
F8. Reporting the best of the last five checkpoints Method caveat
What the paper says. Simulation success is averaged over 100 trials, taking the maximum among the last 5 checkpoints (saved every 500 steps), following the RoboCasa protocol.
What we found. Picking the best of several noisy measurements biases scores upward. In our simulation, with five checkpoints of identical true quality, the reported number is about 3 to 6 points above the truth. The effect applies to the baselines as well, so differences between methods under one protocol are still meaningful, but absolute scores are optimistic.
Why it matters. Compare gaps, not absolute levels, and do not compare with results using a different protocol. Concept: The set-up
Run the example: how much does best-of-five add?
Toy illustrationWhat each part does
- For several true success rates, simulate 20,000 times evaluating 5 checkpoints on 100 trials each.
- Compare the average measured score with the best of the five.
import numpy as np rng = np.random.default_rng(0) trials, checkpoints, repeats = 100, 5, 20000 print("true success rate | average measured | best of 5 checkpoints | inflation") for p in (0.16, 0.26, 0.45, 0.66, 0.90): measured = rng.binomial(trials, p, (repeats, checkpoints)) / trials # each checkpoint evaluated on 100 trials best = measured.max(axis=1).mean() print(f"{p:>17.2f} | {measured.mean():16.3f} | {best:21.3f} | {best - p:+.3f}")Output:
true success rate | average measured | best of 5 checkpoints | inflation 0.16 | 0.160 | 0.203 | +0.043 0.26 | 0.260 | 0.311 | +0.051 0.45 | 0.450 | 0.508 | +0.058 0.66 | 0.660 | 0.715 | +0.055 0.90 | 0.900 | 0.934 | +0.034Assumes five checkpoints of identical true quality and independent noise; real checkpoints differ slightly, so the effect is of this kind but not exactly this size.
-
F9. Few real-robot trials and partial credit Method caveat
What the paper says. Each real task is averaged over about 10 trials (5 for one task), with partial scoring; the pre-trained tests use 15 trials.
What we found. Over the 13 tasks (about 130 trials) a 95% interval is roughly ±8 points; for a single task it is far wider. Some single tasks score higher with 10% of the data than with all of it (for example GR00T on “Dark Cabinet”, 86.6 against 69.7), which is what noise looks like at this sample size.
Why it matters. Overall gaps of 30 points are far outside the noise; claims like “3.8 points lower” or per-task rankings are not. Concept: Real-robot results
-
F10. “Consistently outperforms” is true for averages Method caveat
What the paper says. GR00T N1 outperforms the baselines across all benchmarks and dataset sizes, and performance improves steadily with more data.
What we found. True for each of the nine benchmark averages. Per task it is not: with 30 RoboCasa demonstrations GR00T is behind Diffusion Policy on 10 of 24 tasks, and with 300 DexMG demonstrations behind on 4 of 9. On the GR-1 benchmark GR00T’s average does not improve from 100 to 300 demonstrations (50.0 to 49.3) while the baseline keeps rising (32.7 to 40.4); the authors note that pre-training matters less with very large fine-tuning sets.
Why it matters. Read “consistently” as “on average”. The benefit of pre-training is largest when data is scarce. Concept: Simulation results · example
-
F11. Evidence for generated video is preliminary Method caveat
What the paper says. Neural trajectories give substantial gains; latent actions work better with little data and the IDM with more.
What we found. The headline RoboCasa gains are the better of two label types per regime; real-robot results are for one task group with one label type; the authors call them preliminary. The explanation (the IDM improves with more data) is not tested directly. Video quality is checked by an unnamed commercial judge model on eight frames, with no reported agreement with humans.
Why it matters. Promising, but not yet a measured account of how accurate the generated data is. Concept: Neural trajectories results · concept
-
F12. Missing ablations and unquantified claims Method caveat
What the paper says. The paper describes several design choices as beneficial: cross-attention instead of a mixture of experts, taking the 12th layer, the auxiliary detection loss, the extended data format, and “tightly coupled, jointly trained” modules. It describes the baseline as “confused” and GR00T’s motion as “much smoother”.
What we found. We found no experiment that removes or swaps any of these, and no measurement of confusion or smoothness; they rest on the authors’ observation of rollouts. The only ablation we found of a data source concerns neural trajectories. “End-to-end” also needs care: some parts are frozen (text tokenizer; the language component in post-training).
Why it matters. These are design choices and observations, not demonstrated causes of the results. Concept: System 1 · System 2
-
F13. The pre-trained-model tests have no baseline Method caveat
What the paper says. The pre-trained model reaches 76.6% and 73.3% on two tasks with 15 trials each, showing that large-scale pre-training works.
What we found. No other model is run on these tasks and the sample is small (95% intervals roughly 48 to 91%). The result shows the model is capable on these tasks, not how much pre-training contributed.
Why it matters. A demonstration, not a comparison. Concept: Real-robot results
Limitations the authors state
-
F14. The authors’ own list Limitation
What the paper says. GR00T N1 focuses on short-horizon tabletop manipulation. Long-horizon loco-manipulation needs advances in hardware, architecture and data. A stronger vision-language backbone should improve spatial reasoning and language understanding. Synthetic-data methods struggle to produce diverse, counterfactual and physically accurate data.
What we found. We agree with each; the last one applies directly to the generated videos of Part C and to the sim-to-real gap of simulated data. Long tasks need a planning layer that this model does not provide (see lesson 11.5).
Why it matters. These bound where the reported results should be expected to hold. Concept: Neural trajectories · Simulation data
Knowledge check
Try each question first, then reveal the explanation.
Question 1
In one or two sentences: what are System 1 and System 2 in GR00T N1, and why are they separate?
Explanation
System 2 is a pretrained vision-language model that understands the scene and the instruction (about 10 times a second). System 1 is a Diffusion Transformer that turns noise into a chunk of motor commands (up to 120 actions a second). They are separate because understanding is slow and benefits from broad pretrained knowledge, while control must be fast; they are still trained together.
Question 2
Human videos and generated videos have no robot actions. How does the paper still train on them?
Explanation
It estimates actions: latent actions (a learned summary of what changed between two frames) and, for generated robot videos, pseudo-actions from an inverse dynamics model. These are treated as the actions of separate “robots”.
Question 3
GR00T at 10% of the real data scored 42.6 and Diffusion Policy with all data 46.4. What can and cannot be concluded?
Explanation
The two are comparable within noise (about ±8 points with roughly 130 trials), so the data supports “similar performance with a tenth of the demonstrations”. It does not show that Diffusion Policy is better, and, because the baseline is trained from scratch, it does not show which part of GR00T is responsible.
Question 4
Name one numerical inconsistency and one methodological caveat from the critical review.
Explanation
For example: Table 2 and Table 4 give different DexMG numbers for the same 100-demonstration setting (56.1 / 66.5 against 46.9 / 58.5); and simulation scores report the best of the last five checkpoints, which biases every score upward.
Glossary
- VLA
- Vision-language-action model: images and an instruction in, robot actions out.
- Foundation model
- One large model trained on broad data and then adapted to many tasks.
- Humanoid
- A robot with a human-like body: two arms, hands, a head.
- System 2 / System 1
- The paper’s names for the slower understanding module and the faster action module.
- VLM
- Vision-language model: a model that reads images and text together.
- Token
- A small vector that a transformer reads; images and words are both turned into tokens.
- Cross-attention
- A layer in which one sequence of tokens looks at another (here, actions looking at the scene description).
- DiT
- Diffusion Transformer: a transformer that produces its output by gradual denoising.
- Flow matching
- A way to train a network to move a noisy sample toward a clean one in a few steps.
- Action chunk
- A short sequence of future actions produced at once (16 in GR00T N1).
- Proprioception
- The robot’s own joint readings.
- Embodiment
- One particular robot body: its joints, sensors and grippers.
- MLP
- Multi-layer perceptron: a small, simple neural network.
- Teleoperation
- A person controlling a robot remotely to record a demonstration.
- Retargeting
- Converting a human’s motion into joint commands for a robot.
- Latent action
- A learned code for “what changed between two frames”, used as an action label for video.
- VQ-VAE
- A model that squeezes information through a small codebook of discrete codes.
- IDM
- Inverse dynamics model: from two frames, estimates the actions between them.
- Pseudo-action
- An estimated action label, not what a robot actually executed.
- Neural trajectory
- A robot video generated by a video model and then given pseudo-actions.
- Co-training
- Training on several data sources mixed together.
- Pre-training / post-training
- Broad training on all data first; then fine-tuning on one robot’s data.
- Wilson interval
- A 95% range for a success rate measured over a small number of trials.
- bf16
- A compact 16-bit number format used to speed up large models.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2025). GR00T N1: An open foundation model for generalist humanoid robots. arXiv. arXiv:2503.14734
Cited for:
- Training data is organised as a pyramid of web and human video, synthetic data (simulation and generated video, with latent or inverse-dynamics action labels) and real-robot data; the paper's own table of dataset sizes, hours and evaluation numbers (Tables 2 to 7 and Figure 9) are the numbers re-checked on the decoder page. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
- Architecture details: the Eagle-2 vision-language module (System 2) runs at 10 Hz, the diffusion-transformer action module (System 1) runs at 120 Hz, the action module attends to vision-language tokens by cross-attention, and each embodiment has its own state encoder and action decoder; actions are generated as chunks by flow matching. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
(2023). Sigmoid loss for language image pre-training. ICCV 2023 / arXiv. arXiv:2303.15343
Cited for:
- Proposes a pairwise sigmoid loss for image-text pre-training that operates on image-text pairs and does not need a global view of all pairwise similarities for normalisation, unlike the softmax-normalised contrastive loss used in CLIP-style training. Checked against the paper’s abstract.
(2018). FiLM: Visual reasoning with a general conditioning layer. AAAI 2018 / arXiv. arXiv:1709.07871
Cited for:
- Feature-wise Linear Modulation (FiLM) layers influence a neural network’s computation through a simple feature-wise affine transformation based on conditioning information. Checked against the paper’s abstract.
(2023). Flow matching for generative modeling. ICLR 2023 / arXiv. arXiv:2210.02747
Cited for:
- Flow Matching is a simulation-free way to train continuous normalizing flows by regressing the vector fields of fixed conditional probability paths; sampling works with off-the-shelf numerical ODE solvers. Checked against the paper’s abstract.
(2023). Learning fine-grained bimanual manipulation with low-cost hardware. arXiv:2304.13705
Cited for:
- Introduces Action Chunking with Transformers (ACT), which learns a generative model over sequences of actions; the paper uses action chunking and temporal ensembling to reduce compounding errors. Checked against the paper, via summaries of its text.
(2023). Open X-Embodiment: Robotic learning datasets and RT-X models. arXiv:2310.08864
Cited for:
- Pools more than one million robot trajectories from 22 robot embodiments, assembled by 21 institutions and covering 527 skills. Checked against the paper’s abstract.
(2020). Sim-to-real transfer in deep reinforcement learning for robotics: a survey. IEEE Symposium Series on Computational Intelligence (SSCI) 2020. arXiv:2009.13303 · doi:10.1109/SSCI47803.2020.9308468
Cited for:
- Policies trained in simulation can perform worse on real hardware because simulated and real environments differ (the reality gap); the survey covers domain randomization, domain adaptation, imitation learning, meta-learning and knowledge distillation as responses. Checked against the paper’s abstract.
Go further
- Paper guide: Gemini Robotics (a cloud backbone plus a local decoder)
- Paper guide: RT-2 (the VLA that writes actions as text)
- Foundations: Rectified Flow paper guide
- Lesson 10.1: NVIDIA GR00T N1 and N1.5
- Lesson 4.7: Flow matching for action generation
- Lesson 4.4: Action chunking
- Lesson 8.2: Task success rate and repeatability