π0: A Vision-Language-Action Flow Model for General Robot Control

Part A · about 4 minutes

Start here: the paper at a glance

Part B

Understand the architecture

Follow one decision through the model: a bimanual robot (two arms) is told “fold the shirt”. Plain words first, technical names second; equations wait until Part E.

1. Inputs
2. Vision-language backbone
3. Action expert
4. Flow matching
5. Action chunks

Part C

Understand the training

The paper argues that the recipe matters as much as the architecture. This part covers what data π0 learns from, how very different robots share one model, why there are two training stages, and how language is used.

Part D

Understand the experiments

What was tested, against what, and what the numbers let us conclude. The paper presents its results as bar charts, not tables, so every value here is a reading from a chart and is marked approximate (±0.03 or so). The notes after each table are ours.

Part E · optional

Engineering deep dive

Equations, token layout, attention mask, sizes, timings and scoring rubrics, for readers who want to rebuild or audit the model. Everything is collapsed; Parts A to D do not depend on it.

import numpy as np
rng = np.random.default_rng(0)

s = 0.999
u = rng.beta(1.5, 1.0, 200000)                 # u = (s - tau) / s follows Beta(1.5, 1)
tau = s * (1 - u)

print("average tau:", round(float(tau.mean()), 3), "(exact value:", round(s * (1 - 1.5 / 2.5), 3), ")")
print("share of training samples with tau below 0.5:", round(float(np.mean(tau < 0.5)), 3), "(exact:", round(1 - 0.5 ** 1.5, 3), ")")
edges = np.linspace(0, 1, 6)
counts, _ = np.histogram(tau, bins=edges)
for lo, hi, n in zip(edges[:-1], edges[1:], counts):
    print(f"tau in [{lo:.1f}, {hi:.1f}): {n / len(tau):.3f}")

average tau: 0.399 (exact value: 0.4 )
share of training samples with tau below 0.5: 0.648 (exact: 0.646 )
tau in [0.0, 0.2): 0.286
tau in [0.2, 0.4): 0.250
tau in [0.4, 0.6): 0.211
tau in [0.6, 0.8): 0.164
tau in [0.8, 1.0): 0.089

import numpy as np

# Which token may look at which? Three blocks, as in Appendix B:
#   block 0: image and language tokens (the pre-trained VLM's inputs)
#   block 1: the robot state token
#   block 2: the noisy action tokens
# Rule: inside a block everything sees everything; a block sees earlier blocks, never later ones.
sizes = [4, 1, 3]                                   # tiny toy sizes: 4 image/text tokens, 1 state, 3 actions
block_of = np.repeat(np.arange(len(sizes)), sizes)  # block number of every token: [0 0 0 0 1 2 2 2]

# allowed[i, j] is True when token i may look at token j.
allowed = block_of[None, :] <= block_of[:, None]

names = ["img/text"] * sizes[0] + ["state"] * sizes[1] + ["action"] * sizes[2]
print("row = the token that looks, column = the token it looks at (1 = allowed)")
for i, row in enumerate(allowed.astype(int)):
    print(f"{names[i]:>8}  {''.join(map(str, row))}")

print()
print("image/text tokens see any state or action token:", bool(allowed[:4, 4:].any()))
print("the state token sees any action token:          ", bool(allowed[4, 5:].any()))
print("action tokens see every earlier token:          ", bool(allowed[5:, :5].all()))

row = the token that looks, column = the token it looks at (1 = allowed)
img/text  11110000
img/text  11110000
img/text  11110000
img/text  11110000
   state  11111000
  action  11111111
  action  11111111
  action  11111111

image/text tokens see any state or action token: False
the state token sees any action token:           False
action tokens see every earlier token:           True

import numpy as np
rng = np.random.default_rng(0)       # fixed seed so the numbers are repeatable

d = 18          # action size (all robots are padded to 18 numbers, Section V-A)
w = 8           # width of the action expert in this toy (the real one is 1024)
H = 50          # actions in a chunk

def sinusoidal(tau, width):
    # The flow-matching time tau (a single number) turned into `width` numbers, as in a transformer's position code.
    freqs = 1000.0 ** (-np.arange(width // 2) / (width // 2))
    angles = tau * freqs
    return np.concatenate([np.sin(angles), np.cos(angles)])

def swish(x):
    return x / (1 + np.exp(-x))

# Appendix B: embedding = W3 . swish( W2 . concat( W1 . a, phi(tau) ) )
W1 = rng.normal(size=(w, d)) * 0.1       # action (18 numbers) -> width w
W2 = rng.normal(size=(w, 2 * w)) * 0.1   # joins the action features with the time features
W3 = rng.normal(size=(w, w)) * 0.1       # final mixing

noisy_actions = rng.normal(size=(H, d))  # one noisy action per row
tau = 0.3                                # how far along the noise-to-data path we are

tokens = []
for a in noisy_actions:                  # each action in the chunk becomes one token
    features = np.concatenate([W1 @ a, sinusoidal(tau, w)])
    tokens.append(W3 @ swish(W2 @ features))
tokens = np.array(tokens)

print("noisy action chunk:", noisy_actions.shape, "-> action tokens:", tokens.shape)
print("time encoding for tau = 0.3 has", sinusoidal(tau, w).shape[0], "numbers; it is the same for every action in the chunk")

noisy action chunk: (50, 18) -> action tokens: (50, 8)
time encoding for tau = 0.3 has 8 numbers; it is the same for every action in the chunk

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: places where the paper’s own numbers or equations disagree, where wording is ambiguous, where the method limits what can be concluded, and what the authors list as limits. We looked for these in good faith; none overturns the paper’s main message, and each links back to the concept it affects. Calculations are reproducible in the examples. The figures in this paper are charts, so any statement about a bar is a reading of the chart, accurate to a few hundredths.

Numerical inconsistencies and ambiguities

1. F1. The direction of the flow-matching target and the Euler step Numerical inconsistency

   What the paper says. Noisy actions are τ·A + (1 − τ)·ε, the network is trained to match the “denoising vector field” u = ε − A, and actions are generated by integrating from τ = 0 to τ = 1 with A + δ·v.

   What we found. With that path the actions move in the direction A − ε as τ grows, the opposite of u. Following the printed update with the printed target moves away from the data. This is the same convention GR00T N1 reuses (and its guide flags the same issue).

   Why it matters. Probably a typo or a convention difference between paper and code (the code is the authority). It matters if you re-implement the equations literally; it does not by itself change the reported results. Concept: Flow matching

   Run the example: noise path: variance and direction

   Our calculation

   What each part does

   1. Sample many noise values and build the noisy action at three values of τ.
   2. Compare the measured variance with (1 − τ) and (1 − τ)².
   3. Measure the slope of the path with τ and compare with the printed target.

   import numpy as np
   rng = np.random.default_rng(0)
   
   # Our own measurement of the noisy-action path used in Section IV:
   #   A_tau = tau * A + (1 - tau) * eps,   with eps ~ N(0, 1)
   # The paper writes the path distribution as N(tau * A, (1 - tau) I). If the second argument is a variance,
   # the measured variance below says it should be (1 - tau) squared.
   A = 1.0                                          # one fixed "clean action" (a single number, for simplicity)
   eps = rng.normal(size=1_000_000)
   
   for tau in (0.2, 0.5, 0.8):
       noisy = tau * A + (1 - tau) * eps
       print(f"tau = {tau}: measured variance {noisy.var():.3f} | (1 - tau) = {1 - tau:.3f} | (1 - tau)^2 = {(1 - tau) ** 2:.3f}")
   
   # Direction of travel: how the noisy action moves as tau grows from 0 (noise) to 1 (data).
   a, e = 0.7, -0.4                                  # a clean action and a noise sample (arbitrary values)
   d = 1e-6
   slope = (((0.5 + d) * a + (1 - 0.5 - d) * e) - (0.5 * a + 0.5 * e)) / d      # (A_tau at 0.5 + d minus A_tau at 0.5) / d
   print()
   print("the path moves with slope A - eps =", round(a - e, 3), "as tau increases (measured:", round(slope, 3), ")")
   print("the target printed in the paper is eps - A =", round(e - a, 3), "-> the opposite direction")
   print("an Euler step A + delta * v with v = eps - A therefore moves away from the data")

   Output:

   tau = 0.2: measured variance 0.641 | (1 - tau) = 0.800 | (1 - tau)^2 = 0.640
   tau = 0.5: measured variance 0.250 | (1 - tau) = 0.500 | (1 - tau)^2 = 0.250
   tau = 0.8: measured variance 0.040 | (1 - tau) = 0.200 | (1 - tau)^2 = 0.040
   
   the path moves with slope A - eps = 1.1 as tau increases (measured: 1.1 )
   the target printed in the paper is eps - A = -1.1 -> the opposite direction
   an Euler step A + delta * v with v = eps - A therefore moves away from the data

2. F2. “N(τA, (1 − τ)I)”: variance or standard deviation? Numerical inconsistency

   What the paper says. The probability path is q(A^τ | A) = N(τA, (1 − τ)I), and in practice the noisy actions are A^τ = τA + (1 − τ)ε with ε ~ N(0, I).

   What we found. If the second argument of N is a variance, the construction gives (1 − τ)², not (1 − τ): the measured variance matches the square. If it is meant as a standard deviation the notation is non-standard.

   Why it matters. A notation slip with no effect on the method as implemented (the practical recipe is explicit). Concept: Flow matching · example

3. F3. The attention-head count and the 300-million-parameter expert Ambiguity

   What the paper says. Appendix B gives the Gemma configuration as width 2,048, depth 18, MLP 16,384, num heads 18, 1 key/value head, head size 256, and a downsized action expert of width 1,024 and MLP 4,096, “about 300 million parameters”.

   What we found. Building the expert with Gemma-style layers and the stated sizes gives about 311 million parameters with 8 attention heads (Gemma 2B’s layout, to our knowledge) but about 406 million with 18. So “18 heads” is most likely a typo for 8 (18 is the depth). This is our estimate under the assumptions in the example.

   Why it matters. Minor, but a reader rebuilding the model from the appendix would be misled. Concept: Action expert

   Run the example: can the expert have about 300 million parameters?

   Our calculation

   What each part does

   1. Write the parameter count of one Gemma-style layer from its pieces (attention matrices and a gated MLP).
   2. Fill in the expert’s sizes from Appendix B.
   3. Compare 8 and 18 heads with the stated 300 million.

   # Can we reproduce "about 300 million parameters" for the action expert from the sizes in Appendix B?
   # Assumption: the expert is built like Gemma (gated MLP with 3 matrices, multi-query attention) and
   # interacts with the main model only through attention, so its attention matrices must match the
   # main model's head layout: head_dim 256 and 1 key/value head.
   
   def expert_params(width, mlp_dim, depth, heads, head_dim=256, kv_heads=1):
       q = width * heads * head_dim            # query projection
       kv = 2 * width * kv_heads * head_dim    # key and value projections (one head each)
       o = heads * head_dim * width            # output projection
       mlp = 3 * width * mlp_dim               # gated MLP: gate, up and down matrices
       return depth * (q + kv + o + mlp)
   
   width, mlp_dim, depth = 1024, 4096, 18       # action expert sizes from Appendix B
   
   for heads in (8, 18):
       p = expert_params(width, mlp_dim, depth, heads)
       print(f"with {heads:>2} attention heads the expert has about {p / 1e6:.0f} million parameters")
   print("the paper says: about 300 million; Appendix B lists num heads = 18 for the main model (Gemma 2B has 8)")

   Output:

   with  8 attention heads the expert has about 311 million parameters
   with 18 attention heads the expert has about 406 million parameters
   the paper says: about 300 million; Appendix B lists num heads = 18 for the main model (Gemma 2B has 8)

   Assumes Gemma-style layers, ignores small normalisation weights, and uses the head layout the paper lists.

4. F4. 903 million timesteps against “over 10,000 hours” Ambiguity

   What the paper says. Pre-training uses 903 million timesteps of the authors’ own data (106 million single-arm, 797 million two-armed), and the paper elsewhere says about 10,000 hours of dexterous manipulation data. The appendix says single-arm UR5e and Franka run at 20 Hz and all other robots at 50 Hz.

   What we found. With those rates and the assumption that single-arm means 20 Hz and two-armed means 50 Hz, 903 million timesteps is about 5,900 hours; even at 20 Hz throughout it is about 12,500 hours at most. The numbers fit only if the hours include other data or the timesteps are counted differently (for example after sub-sampling), which the paper does not say.

   Why it matters. The scale claim (“largest robot pre-training mixture”) rests on this figure; the quantity of data is not fully traceable. Concept: The pre-training mixture · example

5. F5. Smaller points Ambiguity

   What the paper says. Figure 3 labels the mobile manipulators “18 DoF” (degrees of freedom); Section V-C gives mobile robots 14-dimensional configurations with 16- or 17-dimensional actions; the padding size of 18 is described as covering two arms, two grippers, a mobile base and a vertically actuated torso.

   What we found. None of the listed robot configurations has a torso, so 18 is a padding size rather than the size of any listed robot. The inference times in Table I add up as stated (14 + 32 + 27 = 73; 73 + 13 = 86).

   Why it matters. No effect on results; worth knowing when reading the figure labels. Concept: Cross-embodiment training

Methodological caveats

1. F6. “Best across the board” is slightly stronger than the charts Method caveat

   What the paper says. π0 generally outperforms other methods; the pre-trained model is frequently better than the non-pre-trained one, sometimes by as much as 2x; on the complex tasks the full recipe performs best across the board.

   What we found. Mostly true, with exceptions visible in the charts: for tupperware in the microwave the 5-hour pre-trained policy is about the same as the baselines and below the from-scratch model; for items in a drawer the pre-trained and scratch curves coincide at 5 and 10 hours; for packing eggs the from-scratch bar is about equal to or above the fine-tuned one; for mobile laundry the zero-shot bar is above the from-scratch one. The paper acknowledges some of these in text.

   Why it matters. Read “generally” literally: pre-training helps most for hard tasks and small data, and not always. Concept: New tasks · Long tasks

2. F7. π0 against π0-small mixes size and pre-training Method caveat

   What the paper says. π0-small (no VLM initialisation) is used to show the benefit of VLM pre-training, for zero-shot results and language following.

   What we found. π0-small is about seven times smaller and differs in six other ways (Part E): another language encoder, an encoder-decoder layout, a smaller image encoder, a non-pre-trained observation transformer and a DiT-style expert. The authors say they cannot remove the size confounder.

   Why it matters. The comparison shows the whole π0 recipe beating this particular smaller model, not the isolated effect of the VLM. Concept: Language · Zero-shot

3. F8. The baselines are not trained like π0 Method caveat

   What the paper says. OpenVLA and Octo are trained on the authors’ mixture for 160 and 320 thousand steps (π0: 700 thousand, parity version 160 thousand); for fine-tuning they start from public checkpoints trained on Open X-Embodiment, while ACT and Diffusion Policy train only on task data.

   What we found. Different pre-training data, steps, model sizes and (for OpenVLA) a no-chunking design all differ between π0 and its baselines. The authors say so, and say they evaluate the specific models rather than the architectures. Hyperparameter tuning for the baselines is not described.

   Why it matters. The results show π0 as a system beats these systems; they do not isolate flow matching, action chunking or the VLM. Concept: The set-up

4. F9. About 10 episodes per task, partial credit, no intervals Method caveat

   What the paper says. Scores are averaged over 10 episodes per task and method, with fractional credit from rubrics that differ from task to task.

   What we found. No confidence intervals or repeated runs are reported. With 10 episodes a single task’s average can move by tenths of a point from one or two episodes, and a per-task rubric of 3 to 12 points means different tasks are not scored on the same scale. Shirt folding is 10 evaluations (5 shirts × 2).

   Why it matters. Large gaps (for example 0.8 against 0.2) are meaningful; small gaps between neighbouring bars, and per-task orderings, are not. Concept: The set-up

5. F10. Design choices with little or no supporting experiment Method caveat

   What the paper says. The paper says separate weights for the action tokens improved performance, hypothesises why a noisy-timestep-heavy Beta distribution suits actions, down-weights tasks by n^0.43 and reports that temporal ensembling hurt.

   What we found. We found no numbers for any of these (no ablation of separate weights, the Beta distribution, the 10 steps, the weighting exponent, or ensembling), and the authors say the best composition of the pre-training data is unknown.

   Why it matters. They are reasonable design decisions and observations, not demonstrated causes of the results. Concept: Action expert · Flow matching · Mixture

6. F11. Open-loop execution limits reaction time Method caveat

   What the paper says. Chunks are executed without re-observation: 0.8 s on the 20 Hz robots and 0.5 s on the 50 Hz robots, and temporal ensembling is not used.

   What we found. Inference takes 73 to 86 ms, so the model is idle most of the time; within a window the robot cannot react to what it sees. This is our reading of the paper’s own numbers and not a measurement of failures. The paper does not report how often failures occur inside a chunk.

   Why it matters. Fast disturbances are handled only at the next replan; shorter windows might help but the paper does not study them. Concept: Action chunks

7. F12. Priority and scale claims Method caveat

   What the paper says. π0 is described as the first flow matching VLA producing high-frequency action chunks for dexterous control, the largest robot pre-training mixture, the longest dexterous tasks in end-to-end robot learning, and a new state of the art.

   What we found. These statements are made “to our knowledge”, and the paper offers no survey that would let a reader verify them. “State of the art” on the complex tasks rests on comparisons with ablations of π0 itself, since the authors report no other method solved them.

   Why it matters. Treat as the authors’ positioning; the evidence in the paper is the experiments above. Concept: Long tasks

Limitations the authors state

1. F13. The authors’ own list Limitation

   What the paper says. The composition of the pre-training data is not understood: which data helps and how to weight it is open. Not all evaluated tasks work reliably, and it is unclear how to predict how much data a task needs. It remains to be seen how much positive transfer there is from combining very diverse data and whether universality extends to other domains such as driving, navigation and legged locomotion.

   What we found. We agree with each; they bound where the results should be expected to hold, and match the gaps in the ablations above.

   Why it matters. The paper presents π0 as a stepping stone and a prototype, not a finished robot foundation model. Concept: Pre-training and post-training · Mixture