VLA · Phase 6 · Modern VLA Models
6.6 — Comparing Architectures, Action Representations, Training Methods and Compute
A side-by-side comparison of the models above across the same dimensions.
Five models, four dimensions. This lesson collects what the earlier lessons established into one table, computes one comparison yourself, and shows how to think about choosing.
Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.
Learning objectives
After this lesson, you should be able to:
- Compare models along architecture, action output, training and compute.
- Count sequential steps per chunk for different outputs.
- Turn requirements into questions.
- Explain why published numbers are not directly comparable.
- List the models and what each states
- Compare architecture and action output
- Compare training data and method
- Estimate compute and latency
- Test on your own task
The one-sentence idea
Compare on stated facts and mark the gaps.
There is no best model, only a best fit.
Four dimensions to compare
The last five lessons described five models. To choose or study among them, compare along the dimensions that the roadmap itself uses: architecture (what the network is built from), action representation (tokens, diffusion, flow), training method (what data, how combined) and compute needs (what hardware to train and run). Here is what the sources in the earlier lessons state, collected in one place, using only facts that were checked:
models = [
# name, architecture (as stated), action output, training data (as stated), size (if stated)
("OpenVLA", "Llama 2 + DINOv2/SigLIP encoder + projector", "256-bin action tokens", "970k real-robot demonstrations", "7B"),
("Octo", "transformer policy", "not stated here", "800k Open X-Embodiment trajectories", "not stated here"),
("pi0", "PaliGemma VLM + action expert", "flow-matching action chunks", "not stated here", "VLM ~3B + expert"),
("pi0.5", "builds on pi0", "flow-matching action chunks", "co-trained heterogeneous data", "not stated here"),
("SmolVLA", "compact VLA", "chunked actions, async inference", "community-collected data", "small (see paper)"),
("RDT-1B", "diffusion transformer", "diffusion action chunks", "multi-robot data, 6K+ bimanual fine-tune", "1.2B"),
]
print(f"{'model':<8} {'action output':<34} {'size'}")
for name, arch, out, data, size in models:
print(f"{name:<8} {out:<34} {size}")Output:
model action output size OpenVLA 256-bin action tokens 7B Octo not stated here not stated here pi0 flow-matching action chunks VLM ~3B + expert pi0.5 flow-matching action chunks not stated here SmolVLA chunked actions, async inference small (see paper) RDT-1B diffusion action chunks 1.2B
Toy model. A summary table we assembled from the earlier lessons. “Not stated here” means the figure was not confirmed from the sources used for this course; look it up in the paper before relying on it.
Note how many cells say “not stated”: an honest comparison has gaps. Be suspicious of comparison tables with no gaps.
Compute at run time: how many sequential steps per chunk?
One dimension you can compute for yourself is how many sequential model steps a method needs to produce a chunk. A token model writes one token per number: dimensions × time steps (lesson 4.2). A diffusion or flow method does a fixed number of denoising or integration steps for the whole chunk (lessons 4.6 and 4.7). FAST-style compressed tokens (lesson 4.8) write far fewer tokens. With assumed settings:
dims, steps = 14, 50 # a 14-number action, 50 steps per chunk
denoise_or_flow_steps = 10 # assumed setting
compressed_tokens = 60 # assumed length of a compressed chunk
print("one token per number: ", dims * steps, "sequential token steps")
print("diffusion or flow, assumed: ", denoise_or_flow_steps, "network passes")
print("compressed tokens, assumed: ", compressed_tokens, "sequential token steps")Output:
one token per number: 700 sequential token steps diffusion or flow, assumed: 10 network passes compressed tokens, assumed: 60 sequential token steps
Toy model. All three settings are assumptions to show how the counts scale. The real values depend on each model’s configuration; and a token step and a full network pass do not cost the same.
The comparison is not simply “fewest steps wins”: token steps are cheap relative to a full denoising pass in some designs, and expensive in others. Measure latency on your hardware (lesson 8.6).
Choosing: questions, not a ranking
There is no best model, only a best fit for a constraint. A tiny decision helper makes the logic explicit. Treat it as a way to organise questions, not as advice:
def suggest(needs_bimanual, tiny_gpu, wants_open_weights_and_tokens, needs_continuous_chunks):
notes = []
if needs_bimanual:
notes.append("look at RDT-1B (built for two-arm manipulation)")
if tiny_gpu:
notes.append("look at SmolVLA (designed for modest hardware)")
if wants_open_weights_and_tokens:
notes.append("look at OpenVLA (open, token actions, LoRA fine-tuning)")
if needs_continuous_chunks:
notes.append("look at pi0 (flow-matching action chunks)")
return notes or ["start by reading the evaluation lessons: define your task first"]
for case in ((True, False, False, False), (False, True, False, False), (False, False, True, False), (False, False, False, False)):
print(case, "->", "; ".join(suggest(*case)))Output:
(True, False, False, False) -> look at RDT-1B (built for two-arm manipulation) (False, True, False, False) -> look at SmolVLA (designed for modest hardware) (False, False, True, False) -> look at OpenVLA (open, token actions, LoRA fine-tuning) (False, False, False, False) -> start by reading the evaluation lessons: define your task first
Before using any model, check the licence, the robot and action definitions it was trained with (lesson 4.1), and whether it was evaluated on something like your task (lesson 8.3).
Caveats for any comparison
Numbers reported in different papers come from different tasks, robots and trial counts, so they are not directly comparable (lessons 3.4 and 5.5). OpenVLA’s claim of 16.5 points over RT-2-X is across 29 tasks and several embodiments[1], a statement about that comparison, not a ranking of all models. The most reliable comparison is one you run yourself on your task with enough trials.
Common misconceptions
Misconception 1 “The biggest model is the best.”
Correction Size trades against memory and latency; fit to the task matters.
Misconception 2 “Reported success rates are comparable across papers.”
Correction Tasks, robots and trial counts differ.
Misconception 3 “A comparison table should have no gaps.”
Correction Gaps are honest where a fact was not confirmed.
Engineering takeaways
- Four dimensions.
- Count steps per chunk.
- Ask questions about constraints.
- Test on your task.
Compare on facts, test on your own task.
Knowledge check
Try each question first, then reveal the explanation.
Question 1
Which models in this phase use flow matching or diffusion for actions?
Explanation
π0 and π0.5 (flow matching), RDT-1B (diffusion).
Question 2
Why can one token per number be slow for long chunks?
Explanation
It needs dimensions × steps sequential token steps.
Question 3
Why not rank models by reported success rates?
Explanation
The numbers come from different tasks, robots and trial counts.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2024). OpenVLA: An open-source vision-language-action model. Conference on Robot Learning (CoRL) 2024. arXiv:2406.09246
Cited for:
- Outperforms the 55B-parameter RT-2-X by 16.5 percentage points of absolute task success across 29 tasks and several robot embodiments, with 7 times fewer parameters. Checked against the paper’s abstract.
What’s next
That completes the modern VLA models. Next: hands-on, starting with the inference pipeline.