6.3 — π0 and π0.5

Learning objectives

After this lesson, you should be able to:

• Describe π0’s two parts and what each does.
• Explain the idea of per-type (expert) weights.
• Estimate whether chunk generation keeps up with execution.
• Describe π0.5’s co-training data mixture.

The one-sentence idea

Understand with a pretrained VLM, act with a flow-matching expert.

Continuous chunks, with the VLM’s knowledge.

π0: a vision-language model with an action expert

π0 (“pi-zero”) is built on top of a pretrained vision-language model so that it inherits Internet-scale semantic knowledge, and it adds a flow-matching architecture for the actions.^[1] The pretrained part is PaliGemma, a vision-language model of about 3 billion parameters made from a SigLIP image encoder and the Gemma-2B language model.^[2] The new part is a separate set of weights for the action outputs, called the action expert, which generates action chunks with flow matching.^[1] Flow matching was the subject of lesson 4.7; chunks were the subject of lesson 4.4. For a guided tour of the whole paper (architecture, training recipe, experiments and a critical review), see the π0 paper guide.

So π0 is a hybrid: language-model machinery for understanding, a continuous generative method for acting. It avoids the token-count problem of lesson 4.2 and the averaging problem of lesson 3.4 at the same time. This was also the setting for FAST: combined with π0, the tokenised variant can scale to 10,000 hours of robot data and match the performance of diffusion VLAs with up to 5× less training time.^[3]

What “separate weights” means

Imagine one long sequence containing both understanding tokens (image patches, words) and action tokens (the noisy action being refined). A single set of weights would have to do both jobs. The “expert” idea gives each kind of token its own weights while all tokens still take part in the same sequence. A small illustration with a token-type-specific linear layer:

import numpy as np
rng = np.random.default_rng(0)

d = 6
W_language = rng.normal(size=(d, d))        # weights used for image and text tokens
W_action   = rng.normal(size=(d, d))        # a separate set of weights for action tokens

types  = ["image", "image", "text", "text", "action", "action", "action"]
tokens = rng.normal(size=(len(types), d))

out = np.zeros_like(tokens)
for i, kind in enumerate(types):
    out[i] = tokens[i] @ (W_action if kind == "action" else W_language)

print("tokens in:", tokens.shape, "| tokens out:", out.shape)
print("action tokens use their own weights:", bool(np.allclose(out[4], tokens[4] @ W_action)))
print("a text token with the action weights would give a different result:", not np.allclose(out[2], tokens[2] @ W_action))

tokens in: (7, 6) | tokens out: (7, 6)
action tokens use their own weights: True
a text token with the action weights would give a different result: True

Toy model. A single linear layer with random weights. In π0 the action expert is a separate set of transformer weights; only the idea of per-type weights is shown here.

The benefit is a division of labour: the pretrained weights stay focused on language and vision knowledge, while the new weights specialise in the continuous, high-frequency action signal.

The cost of generating a chunk

Generating one chunk takes several flow-matching steps (lesson 4.7), each a pass through the network. The arithmetic of lesson 4.6 applies; the numbers below are assumed, not measured, and the actual number of steps and the time per pass should be read from the model’s code:

chunk_steps, control_hz = 50, 50                  # a 50-step chunk at 50 Hz covers one second of motion
flow_steps = 10
pass_ms = 8                                       # assumed time per pass (not a measurement)
one_pass_vlm_ms = 60                              # assumed time for understanding the images and text (not a measurement)

generation_ms = one_pass_vlm_ms + flow_steps * pass_ms
print("time to produce a chunk:", generation_ms, "ms")
print("chunk covers:", chunk_steps / control_hz * 1000, "ms of motion")
print("model busy for", round(100 * generation_ms / (chunk_steps / control_hz * 1000), 1), "% of the time it takes to execute the chunk")

time to produce a chunk: 140 ms
chunk covers: 1000.0 ms of motion
model busy for 14.0 % of the time it takes to execute the chunk

Toy model. Every number here is an assumption chosen to make the arithmetic readable.

As long as producing the next chunk takes less time than executing the current one, the robot can keep moving smoothly. If it does not, the robot has to pause or run open-loop; lessons 8.6 and 9.1 return to this.

π0.5: from the lab to new homes

π0.5 builds on π0 and is trained by co-training on heterogeneous data: data from multiple robots, high-level semantic prediction, web data and other sources, with multimodal training examples that mix images, language commands, object detections, subtask prediction and low-level actions.^[4] The authors report that this kind of knowledge transfer is essential for effective generalisation, and describe the first end-to-end learning-enabled system performing long-horizon, dexterous tasks such as cleaning a kitchen or bedroom in entirely new homes.^[4]

The lesson for training recipes is the one from lesson 5.4: more kinds of data, mixed together, help a model generalise. A toy sampler shows what “mixing” means in practice:

import numpy as np
rng = np.random.default_rng(0)

sources = {"robot demonstrations (many robots)": 0.5, "web image-text": 0.2, "subtask / semantic labels": 0.2, "object detections": 0.1}
names, probs = list(sources), list(sources.values())

draws = rng.choice(len(names), size=10_000, p=probs)
for i, name in enumerate(names):
    print(f"{name:<38} {np.mean(draws == i):.3f} of training examples")

robot demonstrations (many robots)     0.499 of training examples
web image-text                         0.201 of training examples
subtask / semantic labels              0.202 of training examples
object detections                      0.098 of training examples

Toy model. The sampling weights are made up; π0.5’s actual mixture is described in its paper.

Whether a model works in a new home is, as lesson 5.5 argued, a claim to read carefully: which kinds of generalisation, how many homes, how many trials.

Common misconceptions

Engineering takeaways

1. PaliGemma plus an action expert.
2. Flow matching generates chunks.
3. Keep generation faster than execution.
4. π0.5 mixes many data sources.

π0: a VLM that understands, an expert that moves.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: SmolVLA, a compact VLA for modest compute.

Back to the VLA roadmap