VLA · Paper guide

OpenVLA: An Open-Source Vision-Language-Action Model

The open vision-language-action model that made RT-2’s idea something a lab can download, finetune and serve, explained through the questions an engineer would ask.

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al.
arXiv:2406.09246 (v3)
Overview: about 4 minutes
Full guide: about 55 minutes

This is a guide to the OpenVLA paper, not a sentence-by-sentence paraphrase. OpenVLA is the open counterpart of the idea in RT-2: a vision-language model that writes robot actions as text. It sits between Octo, which is a small policy trained on robot data only, and the flow-matching VLAs in the other guides, which replace the token output with a continuous action generator. Part A is a 4-minute overview. Parts B to D explain the method, the data and the experiments one idea at a time. Equations and tables are folded away in Part E, and Part F is a critical review of the paper’s numbers and claims.

Start with the overview Jump to the critical review

How to use this page

In a hurry: read Part A (about 4 minutes).
To understand the method: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example, then technical details and where to find them in the paper.
To check or rebuild it: open the folded sections (the tokeniser, hyperparameters, the mixture, runnable examples) and read Part F.

Open the paper (arXiv:2406.09246) beside this page; section, table and figure references are to it. We paraphrase and never reproduce its text. Statements are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own small model of an idea, and are never presented as results of the paper.

Part A · about 4 minutes

Start here: the paper at a glance

The problem

Vision-language-action (VLA) models take a pretrained vision-language model (VLM) and teach it to output robot actions. RT-2 showed the idea works, but it was closed: the weights, the training data mixture and the training code were not available, and nobody had shown how to adapt such a model to a new robot with ordinary hardware. The authors argue that robotics needs what the language-model world already has, an open model that anyone can download, finetune and serve.

The main contribution

OpenVLA, a 7-billion-parameter open VLA. A Prismatic-7B vision-language model, which pairs two image encoders with a Llama 2 language model, is finetuned on 970 thousand robot episodes from Open X-Embodiment. Weights, code and training pipeline are released.^[1]
Actions as text tokens. Each of the seven action numbers is cut into 256 bins and written as one of 256 rarely used words of the language model; the model is trained to predict them with the ordinary next-word loss.^[1]
Evidence on use, not only on accuracy. Direct tests on two robots against RT-1-X, Octo and the 55-billion-parameter RT-2-X, finetuning on new Franka robot tasks against Diffusion Policy and Octo, and experiments on low-rank adaptation (LoRA) and 4-bit serving that bring the cost down to one consumer-grade graphics card.^[1]

How the model works, in one paragraph

One camera image goes through two pretrained vision networks, SigLIP and DINOv2, whose outputs are joined; a small two-layer multilayer perceptron (MLP) projects the result into the language model’s input space. The text prompt (“What should the robot do to put eggplant in bowl?”) is turned into tokens as usual. The Llama 2 model then writes seven tokens, one at a time. Each token is mapped back to a number, and the seven numbers are the change in hand position, the change in hand rotation and the gripper command, rescaled to the robot’s units. There is no separate action network: everything the policy knows sits in the language model and the two image encoders, and all of it, including the image encoders, is trained on robot data.

What it learns from, and what it is tested on

Learns from: 970 thousand episodes, mostly from datasets that Octo also used, with a few additions such as the DROID dataset at a 10% weight (removed for the last third of training). Tested on: two robots used as they are (a WidowX arm in a Bridge sink scene, 17 tasks with 10 trials each, and the Google mobile manipulator, 12 tasks with 5 trials each); a Franka arm finetuned on seven tasks with 10 to 150 demonstrations each; a comparison of finetuning methods and of numeric precision; and the LIBERO simulation benchmark in an appendix.

What the paper reports

Section 5 and Tables 1, 2, 4, 6 and 7, as printed.
Test	OpenVLA	Comparison
Bridge WidowX, 170 rollouts (success, %)	70.6	50.6 RT-2-X; 20.0 Octo; 18.5 RT-1-X
Google robot, 60 rollouts (%)	85.0	78.3 RT-2-X; 33.3 RT-1-X; 26.7 Octo
Franka finetuning, 99 and 30 rollouts (%)	67.2 and 58.3	Diffusion Policy 48.5 and 35.0; Octo 43.4 and 38.3
Finetuning methods, 33 rollouts (%)	68.2 with LoRA (1.4% of weights); 69.7 full	47.0 with a frozen image encoder; 30.3 last layer only
4-bit serving, 80 rollouts (%)	71.9, in 7.0 GB	71.3 in bfloat16 (16.8 GB)

What to keep in mind

The headline “16.5% better than RT-2-X” is 20 points on the Bridge robot and 6.7 on the Google robot, where the difference is within chance. RT-2-X is better on the semantic tasks.
Per task there are 5 to 10 trials, and every number is one run. The large gaps (against RT-1-X, Octo, scratch training) hold; many small ones, such as LoRA against full finetuning, cannot be told apart from noise.
Diffusion Policy is better on narrow single-instruction tasks; OpenVLA’s advantage is on tasks with several objects and language, and there it is level with a finetuned Octo.
The LoRA and quantisation experiments use a smaller variant of the model, not the released one.
Several printed numbers disagree with each other (Part F).

If you read nothing else: OpenVLA shows that a 7-billion-parameter vision-language model, finetuned end to end on robot episodes with actions written as text tokens, is a stronger open starting point than earlier open policies, and can be adapted and served on one consumer-grade card; the evidence is real but rests on small trial counts, one run per number, and comparisons that differ in more than the idea being credited.

Check your understanding: Why does OpenVLA have no separate action network?

Because actions are written as ordinary tokens of the language model’s vocabulary, so the same network and the same next-token loss handle both text and actions. The price is that the model writes seven tokens one after another for every action.

Part B

Understand the method

OpenVLA’s architecture is deliberately plain: a vision-language model whose vocabulary has been taught to say robot actions. The ideas worth understanding are why two image encoders, how an action becomes tokens, and what is trained. Each idea below follows the same pattern, from intuition to where to find it in the paper.

One action. Orange marks the parts the guide calls out as the paper’s main design choices. Drawn by us from Section 3 and Figure 2 of the paper.

1. The starting point: a vision-language model

Intuition: Begin with a network that already reads pictures and writes sentences, and teach it a new vocabulary.
The problem it solves: Robot datasets are small (100 thousand to a million episodes) next to the web-scale data that taught vision and language models to recognise objects and follow phrases. Training a policy from scratch throws that knowledge away.
How it works: The backbone is Prismatic-7B, a vision-language model with three parts: a visual encoder (about 600 million parameters) that turns the image into patch features, a small projector (a two-layer MLP) that maps them into the language model’s input space, and a 7-billion-parameter Llama 2 language model. Prismatic was itself finetuned from its parts on a mixture of about one million image-text and text-only samples (the LLaVA 1.5 mixture). OpenVLA starts from this model and finetunes all of it for robot actions.
Example: Illustration, not from the paper Asked “What should the robot do to put eggplant in bowl?”, the pretrained model could describe the picture. After robot finetuning, the same network answers with seven action tokens instead.
Why the authors chose it: The authors give three reasons: the image and language parts are already aligned by web-scale training, a generic architecture lets them reuse the machinery built for language-model training, and robotics inherits future improvements in vision-language models.
Technical details: The training data of SigLIP, DINOv2 and Llama 2 are not public, so “open” here means the weights of the parts and the authors’ own data, code and checkpoints. The authors say so in a footnote.
Evidence in the paper: §3.1 · Figure 2 · §2

Check your understanding: Which parts of OpenVLA come from the web, and which from robots?

SigLIP, DINOv2 and Llama 2 are pretrained on web data, and Prismatic aligns them on about one million image-text samples. Everything after that, including the image encoders, is trained on 970 thousand robot episodes.

2. Two image encoders instead of one

Intuition: One network is good at what things are, the other at where they are; give the model both.
The problem it solves: Common vision-language models use a single encoder such as CLIP or SigLIP, trained to match images with captions. Such features describe the scene well but may lose the fine spatial detail that reaching and grasping need.
How it works: Prismatic passes each image patch through both SigLIP and DINOv2, a network trained without captions that is known for spatial features, and joins the two feature vectors channel by channel. The authors credit the DINOv2 half with better spatial reasoning and hence better robot control, and in their early tests a fused-backbone policy beat a LLaVA-based one by about 10 points of success.
Example: Illustration, not from the paper For the instruction “put the red bottle in the pot”, SigLIP helps recognise “red bottle”; DINOv2 helps place the gripper above it.
Why the authors chose it: Spatial detail is the argument for the extra encoder; the ablation in Part D asks how much it matters.
Technical details: The ablation (Table 9) removes DINOv2 from a Bridge-only model and loses 5 points on eight tasks, which an exact test cannot distinguish from chance (Part F). The 10-point comparison with LLaVA is given in the text only.
Evidence in the paper: §3.1 · §3.4 (VLM backbone) · Appendix D.2 (Table 9)

Check your understanding: What is the evidence that the fused encoder matters?

A roughly 10-point gain over a LLaVA-based policy (text only) and a 5-point gain on eight tasks in Table 9. The first compares different pretrained models in more than the encoder; the second is within chance.

3. Writing an action as seven tokens

Intuition

Turn each action number into a word the language model can already say.

The problem it solves

A language model outputs tokens from a fixed vocabulary, not real numbers. To use it as a policy, every action must become a sequence of tokens and back.

How it works

The action has seven numbers: change in position (3), change in rotation (3) and the gripper. Each number is cut into 256 equal bins covering the range between the 1st and 99th percentile of that number in the training data; a value outside the range is clipped. The bin index (0 to 255) becomes a token. The Llama tokenizer reserves only 100 spare tokens, so the 256 action tokens overwrite the 256 least-used tokens of the vocabulary. At run time the model writes seven tokens; each is mapped to the centre of its bin and rescaled to the robot’s units.

Example

Illustration, not from the paper Seven bins [12, 200, 255, 0, 17, 128, 255] become the token ids 31756, 31944, 31999, 31744, 31761, 31872, 31999 in a 32,000-word vocabulary.

Why the authors chose it

It needs no change to the model or the training code. The authors choose percentiles over the minimum and maximum, which RT-2 used, so a few extreme actions do not stretch the bins.

Technical details

Toy illustration The example makes up 5,000 small movements and 3 outliers, builds the bins both ways and prints the bin width and the error. With min-max bounds, the 256 bins are 20 times wider and only 23 are ever used; with percentiles all 256 are used and the outliers are clipped to the edge of the range.

Run the example: quantile bins and the overwritten vocabulary

Toy illustration

What each part does

Make toy actions with a few outliers.
Build 256 bins from the 1st to 99th percentile and from min to max; compare width and error.
Show the token ids that the last 256 vocabulary entries become.

import random
# Toy: turn one continuous action dimension into 256 integer tokens, the way Section 3.2 describes (bins between the 1st and 99th quantile),
# and compare with min-max bounds when a few outliers exist. Made-up data, no model.
rng = random.Random(0)
actions = [rng.gauss(0.0, 0.01) for _ in range(5000)] + [0.5, -0.4, 0.45]   # 5,000 small moves and 3 outliers
xs = sorted(actions)
q = lambda p: xs[int(p * (len(xs) - 1))]
bounds = {"1st to 99th quantile (OpenVLA)": (q(0.01), q(0.99)), "min to max (RT-2 style)": (xs[0], xs[-1])}
def to_token(a, lo, hi, bins=256):
    a = min(max(a, lo), hi)                          # values outside the range are clipped
    return min(bins - 1, int((a - lo) / (hi - lo) * bins))
def from_token(t, lo, hi, bins=256):
    return lo + (t + 0.5) * (hi - lo) / bins         # centre of the bin
for name, (lo, hi) in bounds.items():
    width = (hi - lo) / 256
    typ = [a for a in actions if lo <= a <= hi]
    err = sum(abs(from_token(to_token(a, lo, hi), lo, hi) - a) for a in typ) / len(typ)
    used = len({to_token(a, lo, hi) for a in actions})
    print(f"{name:32s} range [{lo:+.4f}, {hi:+.4f}]  bin width {width:.5f}  mean error {err:.5f}  bins used {used}")
lo, hi = bounds["1st to 99th quantile (OpenVLA)"]
print(f"outlier 0.5 is clipped to {from_token(to_token(0.5, lo, hi), lo, hi):+.4f} (the price of ignoring outliers)")
print()
# Where the 256 action tokens live: the last 256 ids of a 32,000-token vocabulary (Llama 2 size, as used by the paper's backbone).
vocab = 32000
first = vocab - 256
print(f"token ids {first} to {vocab - 1} are overwritten; action bin b becomes token id {first} + b")
print("7 dimensions -> 7 tokens per step; for the bins [12, 200, 255, 0, 17, 128, 255]:", [first + b for b in (12, 200, 255, 0, 17, 128, 255)])

Output:

1st to 99th quantile (OpenVLA)   range [-0.0230, +0.0237]  bin width 0.00018  mean error 0.00005  bins used 256
min to max (RT-2 style)          range [-0.4000, +0.5000]  bin width 0.00352  mean error 0.00088  bins used 23
outlier 0.5 is clipped to +0.0236 (the price of ignoring outliers)

token ids 31744 to 31999 are overwritten; action bin b becomes token id 31744 + b
7 dimensions -> 7 tokens per step; for the bins [12, 200, 255, 0, 17, 128, 255]: [31756, 31944, 31999, 31744, 31761, 31872, 31999]

Made-up numbers and no model; it illustrates the design, not OpenVLA’s measurements.

Evidence in the paper

§3.2 · Figure 2

Check your understanding: What does ignoring outliers cost?

Actions beyond the 1st or 99th percentile are clipped to the edge of the range, so the model can never ask for them. The paper accepts this to keep the bins fine for the typical range.

4. What is trained, and how the policy runs

Intuition: Treat the robot as a language task: given the picture and the question, predict the answer one word at a time.
The problem it solves: The model has to be trained so that the seven tokens depend on the image and the instruction, without special machinery for robots.
How it works: Training is the usual next-token prediction with cross-entropy loss, evaluated on the action tokens only. All parameters, including the image encoders, are updated. At run time the model receives the current image and instruction, writes seven tokens, and the robot executes the decoded action; this repeats at every step. There is no action chunking, no history, no proprioception and one camera image.
Example: Illustration, not from the paper At 6 steps a second the robot asks the model for a new seven-token answer every 167 milliseconds, and every answer is based on the latest image alone.
Why the authors chose it: A single-image, single-step policy is the simplest form of the idea. The authors list the missing inputs and chunking as limitations and future work.
Technical details: The design notes (Section 3.4) report that the image encoder should be finetuned and not frozen, that training should continue for 27 epochs, until the accuracy of the action tokens exceeds 95%, with a fixed learning rate of 2e-5 and no warm-up, and that 224 by 224 pixel images are enough.
Evidence in the paper: §3.2 · §3.4 · §6

Check your understanding: Why is the loss computed on the action tokens only?

The prompt is given, not predicted; the only part the robot needs the model to get right is the action, so the loss scores only those seven tokens.

Part C

Understand the data and the training

The data recipe follows Octo, with a few additions; what the paper adds is a record of which design choices mattered in small experiments on Bridge data, and the cost of the final run. The numbers in this part are the paper’s, and the cells re-check the arithmetic.

The data: 970 thousand episodes from Open X-Embodiment

Intuition

Pool many labs’ demonstrations and re-balance them.

The problem it solves

A generalist policy needs many embodiments, scenes and tasks, but the raw Open X-Embodiment collection (more than 70 datasets, more than 2 million episodes at the time) mixes input and output formats and has uneven quality.

How it works

The authors keep only manipulation datasets with at least one third-person camera and single-arm end-effector control. They reuse the mixture weights of Octo, which down-weights repetitive datasets and up-weights diverse ones, and add a few newer datasets, among them DROID at a weight of 10%. The table lists 27 datasets and the total is 970 thousand episodes, against 800 thousand for Octo and 350 thousand for RT-2-X.

Example

Illustration, not from the paper A batch draws about 13% of its examples from Bridge, 12.7% each from Fractal and Kuka, 10% from DROID and 7.5% from BC-Z.

Why the authors chose it

The authors credit part of OpenVLA’s advantage over RT-2-X to the larger and more carefully cleaned dataset, and test the effect of Open X-Embodiment pretraining in an ablation.

Technical details

Our calculation By our arithmetic the listed shares add up to 94.9%, not 100%. The three largest datasets make up 38.7% (Octo’s were 51.0%), the top five 56.2%, and ten of the 27 datasets are below 1% (4.4% together). DROID was removed for the last third of training and its weight redistributed, so it averaged about 6.7%.

Run the example: the training mixture

Our calculation

What each part does

Type in the shares of Table 3.
Add them up and compute the share of the largest datasets.
Work out DROID’s average share given that it was dropped for the last third.

# Table 3: OpenVLA training mixture (percent). "<0.1" is entered as 0.05.
mix = {"Fractal": 12.7, "Kuka": 12.7, "Bridge": 13.3, "Taco Play": 3.0, "Jaco Play": 0.4, "Berkeley Cable Routing": 0.2, "Roboturk": 2.3, "Viola": 0.9, "Berkeley Autolab UR5": 1.2, "Toto": 2.0,
 "Language Table": 4.4, "Stanford Hydra": 4.4, "Austin Buds": 0.2, "NYU Franka Play": 0.8, "Furniture Bench": 2.4, "UCSD Kitchen": 0.05, "Austin Sailor": 2.2, "Austin Sirius": 1.7, "DLR EDAN": 0.05,
 "IAMLab CMU Pickup Insert": 0.9, "UTAustin Mutex": 2.2, "Berkeley Fanuc": 0.7, "CMU Stretch": 0.2, "BC-Z": 7.5, "FMB": 7.1, "DobbE": 1.4, "DROID": 10.0}
tot = sum(mix.values()); srt = sorted(mix.items(), key=lambda kv: -kv[1])
print(f"datasets: {len(mix)}   shares add up to {tot:.2f}%")
for k in (3, 5, 10):
    print(f"  top {k:2d} datasets: {sum(v for _, v in srt[:k]):5.1f}%   ({', '.join(n for n, _ in srt[:k])})")
print(f"  datasets under 1%: {sum(v < 1 for v in mix.values())}, together {sum(v for v in mix.values() if v < 1):.2f}%")
print()
print("DROID: 10% of the mixture, but removed for the last third of training and its weight redistributed:")
share = (2 / 3) * 10.0
print(f"  average DROID share over the run, if training time is split 2/3 and 1/3: {share:.1f}% of the batches (3.3 points less than the listed 10%)")
oct_ = {"Fractal": 17.0, "Kuka": 17.0, "Bridge": 17.0}
print("  Octo's three largest datasets were 17% each (51.0%); here:", ", ".join(f"{n} {mix[n]}%" for n in ("Fractal", "Kuka", "Bridge")), f"= {sum(mix[n] for n in oct_):.1f}%")
print()
print("scale: 970 thousand episodes in this paper against 350 thousand for RT-2-X, 800 thousand for Octo:")
print(f"  {970 / 350:.2f} times RT-2-X; {970 / 800:.2f} times Octo; Open X-Embodiment has more than 2 million episodes, so the mixture uses less than half of it ({970 / 2000:.0%} at most)")

Output:

datasets: 27   shares add up to 94.90%
  top  3 datasets:  38.7%   (Bridge, Fractal, Kuka)
  top  5 datasets:  56.2%   (Bridge, Fractal, Kuka, DROID, BC-Z)
  top 10 datasets:  77.5%   (Bridge, Fractal, Kuka, DROID, BC-Z, FMB, Language Table, Stanford Hydra, Taco Play, Furniture Bench)
  datasets under 1%: 10, together 4.40%

DROID: 10% of the mixture, but removed for the last third of training and its weight redistributed:
  average DROID share over the run, if training time is split 2/3 and 1/3: 6.7% of the batches (3.3 points less than the listed 10%)
  Octo's three largest datasets were 17% each (51.0%); here: Fractal 12.7%, Kuka 12.7%, Bridge 13.3% = 38.7%

scale: 970 thousand episodes in this paper against 350 thousand for RT-2-X, 800 thousand for Octo:
  2.77 times RT-2-X; 1.21 times Octo; Open X-Embodiment has more than 2 million episodes, so the mixture uses less than half of it (48% at most)

Arithmetic on Table 3; “<0.1” is entered as 0.05.

Evidence in the paper

§3.3 · Appendix A (Table 3)

Check your understanding: Why did the authors remove DROID for the last third of training?

The accuracy of the action tokens on DROID stayed low, which suggested that the model or the weight was too small for its diversity; they did not want it to harm the quality of the final model.

Cleaning the data: why a policy can freeze

Intuition: A model that imitates every recorded action also imitates doing nothing.
The problem it solves: In the BridgeData V2 dataset the first action of every demonstration is all zeros. A highly expressive model that imitates the data learns to output the zero action, and at run time it stops moving.
How it works: The authors drop the first transition of every Bridge demonstration, which fixes the freezing in most cases. In the LIBERO appendix they drop all near-zero (no-op) actions for the same reason. RT-2-X was trained without this cleaning, so for Bridge tests the authors, like the Open X-Embodiment authors, ask it for its second most likely action, because the most likely one is often all zeros.
Example: Illustration, not from the paper If every demonstration starts with one no-op action, a policy that copies this can start by predicting “stay still” and never leave that state.
Why the authors chose it: It is a practical lesson for anyone training a policy from logged data, and part of the explanation the authors give for the large gap with RT-2-X on the Bridge robot.
Technical details: Appendix C also says that re-querying RT-2-X only when the first answer was all zeros worked worse than always using the second-best, which the authors attribute to pauses in the robot’s motion. The comparison therefore uses a workaround for RT-2-X that is not how its designers meant it to be used (Part F).
Evidence in the paper: Appendix C · Appendix E.1 · §5.1

Check your understanding: Why does a frozen robot show up as a policy problem and not a data problem?

The policy’s own output is “no movement”, so from outside it looks like the model has stopped; the cause is that zero actions in the training data are a legitimate answer to match.

The training recipe: what small experiments decided

Intuition

A handful of cheap trials on Bridge data settled the expensive choices.

The problem it solves

The final run is too costly to repeat, so choices about backbone, resolution and training length have to come from smaller experiments.

How it works

Design experiments used Bridge data only. Prismatic beat LLaVA and IDEFICS-1 as backbone, mostly on tasks with several objects. The authors found no gain from 384-pixel over 224-pixel images, which take about three times as long to train. They found that finetuning the image encoder was necessary, in contrast to what is usual for vision-language models. Real-robot performance kept improving until the action-token accuracy passed 95%, which took 27 passes over the data. A fixed learning rate of 2e-5, the one used in pretraining, worked best with no warm-up.

Example

Illustration, not from the paper An engineer who freezes the image encoder because “that is what the vision-language literature says” would, according to Table 10, lose about 33 points in this setting.

Why the authors chose it

These are the choices another lab would otherwise have to rediscover.

Technical details

Our calculation Under the assumption of 14-pixel patches (not stated in the paper), the 384-pixel sequence is 2.6 times longer than the 224-pixel one, which fits “about 3 times longer to train”. The final run used 64 A100 graphics cards for 14 days, which is 21,504 card-hours, against the 21,500 stated.

Run the example: cost, speed and resolution arithmetic

Our calculation

What each part does

Check 64 cards x 14 days against 21,500 card-hours.
Estimate the sequence length at 224 and 384 pixels.
Work out how many tokens per second the 6 Hz rate means, and what a 50 Hz robot would need.

# Cost, size and speed arithmetic from Sections 3.4, 3.5 and 5.4.
gpus, days = 64, 14
print(f"pretraining: {gpus} A100 GPUs x {days} days x 24 h = {gpus * days * 24:,} GPU-hours (paper: 21,500); batch size 2,048, 27 epochs")
eps, epochs, batch = 970_000, 27, 2048
print(f"  27 passes over 970 thousand episodes; the number of training samples is not stated, so steps cannot be computed from the paper")
print()
print("input resolution: 224 x 224 against 384 x 384 ('the latter takes 3x longer to train').")
print("  assumption not in this paper: the vision encoders cut the image into 14 x 14 pixel patches, so 16 x 16 = 256 patches at 224 and 27 x 27 = 729 at 384")
for res, p in ((224, 16 * 16), (384, 27 * 27)):
    prompt = 25     # a rough length for the question text; any value of this size gives the same picture
    n = p + prompt + 7 + 1
    print(f"  {res} px: {p} image tokens, about {n} tokens in the sequence")
a, b = 256 + 25 + 8, 729 + 25 + 8
print(f"  sequence ratio {b / a:.2f}: a cost that grows in step with the sequence gives {b / a:.1f} times; the quadratic part of attention pushes it higher, which fits the paper's 3x")
print()
print("inference")
print(f"  6 Hz on an RTX 4090 means one 7-token action every {1000 / 6:.0f} ms, about {6 * 7} generated tokens per second, and each token needs a full pass through 7 billion parameters")
print(f"  bfloat16 weights: 7.19 billion x 2 bytes = {7.1881 * 2:.1f} GB; 15 GB quoted in Section 3.5, 16.8 GB in Table 2")
print(f"  reading all the weights once per token at 1,000 GB/s (an assumed 4090-class memory speed) would take {7.1881 * 2 / 1000 * 1000:.1f} ms; 7 tokens: {7 * 7.1881 * 2:.0f} ms of the {1000 / 6:.0f} ms budget")
print("  (the estimate ignores the image encoders and attention; it only shows why generating 7 tokens one at a time is slow)")
print()
print("control rates")
for name, hz in (("OpenVLA on RTX 4090", 6.0), ("BridgeData V2 controller", 5.0), ("DROID controller", 15.0), ("ALOHA (named in the limitations)", 50.0)):
    print(f"  {name:34s} {hz:5.1f} Hz  {1000 / hz:6.1f} ms per action")
print(f"  with 7 tokens per action and no chunking, ALOHA's 50 Hz would need {50 * 7} generated tokens per second, {50 * 7 / (6 * 7):.1f} times what the 4090 does")

Output:

pretraining: 64 A100 GPUs x 14 days x 24 h = 21,504 GPU-hours (paper: 21,500); batch size 2,048, 27 epochs
  27 passes over 970 thousand episodes; the number of training samples is not stated, so steps cannot be computed from the paper

input resolution: 224 x 224 against 384 x 384 ('the latter takes 3x longer to train').
  assumption not in this paper: the vision encoders cut the image into 14 x 14 pixel patches, so 16 x 16 = 256 patches at 224 and 27 x 27 = 729 at 384
  224 px: 256 image tokens, about 289 tokens in the sequence
  384 px: 729 image tokens, about 762 tokens in the sequence
  sequence ratio 2.64: a cost that grows in step with the sequence gives 2.6 times; the quadratic part of attention pushes it higher, which fits the paper's 3x

inference
  6 Hz on an RTX 4090 means one 7-token action every 167 ms, about 42 generated tokens per second, and each token needs a full pass through 7 billion parameters
  bfloat16 weights: 7.19 billion x 2 bytes = 14.4 GB; 15 GB quoted in Section 3.5, 16.8 GB in Table 2
  reading all the weights once per token at 1,000 GB/s (an assumed 4090-class memory speed) would take 14.4 ms; 7 tokens: 101 ms of the 167 ms budget
  (the estimate ignores the image encoders and attention; it only shows why generating 7 tokens one at a time is slow)

control rates
  OpenVLA on RTX 4090                  6.0 Hz   166.7 ms per action
  BridgeData V2 controller             5.0 Hz   200.0 ms per action
  DROID controller                    15.0 Hz    66.7 ms per action
  ALOHA (named in the limitations)    50.0 Hz    20.0 ms per action
  with 7 tokens per action and no chunking, ALOHA's 50 Hz would need 350 generated tokens per second, 8.3 times what the 4090 does

Arithmetic on Sections 3.4, 3.5 and 6; the patch size and memory speed are our assumptions, marked in the output.

Evidence in the paper

§3.4 · §3.5 · Appendix D.3 (Table 10)

Check your understanding: Which of these design decisions is supported by a table and which only by the text?

The frozen-against-finetuned encoder has Table 10; the backbone comparison, the resolution result, the epochs and the learning rate are stated in the text without numbers.

Part D

Understand the experiments

Four questions: does OpenVLA work out of the box on several robots, does it finetune well to a new one, can the cost of finetuning and serving come down, and which parts of the recipe matter. This part explains each test and what the tables say. Our re-calculations are in the folded examples.

The set-up: robots, tasks, comparisons and trial counts

How it works

Two robots are used directly. The WidowX arm in the BridgeData V2 sink scene gets 17 tasks with 10 trials each, in five groups: visual, motion, physical and semantic generalisation, and language grounding (the prompt names one of two objects). The Google mobile manipulator gets 12 tasks with 5 trials each, five in-distribution and seven out-of-distribution. For finetuning, a Franka arm is tested on six tabletop tasks (three narrow, with one instruction, and three with several objects and instructions) and one Franka-DROID task, with 5 or 6 trials for the out-of-distribution variants and 10 to 18 for the others. Comparisons are RT-1-X (35 million parameters), Octo (93 million) and RT-2-X (55 billion, used through a restricted interface) for the direct tests, and Diffusion Policy, a matched version of it, Octo and an OpenVLA trained from the base vision-language model alone for finetuning. Partial credit of 0.5 is given on several tasks, so counts can be fractions.

Technical details

Our calculation With 5 trials a success rate near 50% has a standard error of 22 points, with 10 trials 16, with 33 trials 9, with 80 trials 6 and with 170 trials 4. An exact test calls two rates different only when they are at least 50 points apart with 10 trials per method, and 18 with 80.

Run the example: how much a success rate can move by chance

Our calculation

What each part does

Take the trial counts used in the paper: 5 to 170.
Compute the standard error of a rate near 50%.
Find the smallest gap an exact two-sided test treats as different, and test the headline gaps.

import math
# How much can a success rate move by chance? Trial counts used across the OpenVLA paper.
def se(p, n): return 100 * math.sqrt(p * (1 - p) / n)
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
print("trials | where in the paper                                       | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)")
for n, where in ((5, "Google robot, per task; Franka OOD rows"), (10, "Bridge, per task; Franka in-distribution rows"), (33, "Table 1, per strategy"), (60, "Google robot, per method; Table 10"), (80, "Tables 2, 5, 9, 11"), (99, "Franka-Tabletop, per method"), (129, "all Franka rollouts"), (170, "Bridge, per method")):
    base = n // 2
    gap = next((100 * (k - base) / n for k in range(base, n + 1) if fisher(k, n, base, n) < 0.05), None)
    print(f"{n:6d} | {where:56s} | {se(0.5, n):9.1f} pts | {gap:.0f} points" if gap is not None else f"{n:6d} | {where:56s} | {se(0.5, n):9.1f} pts | none")
print()
print("headline gaps in the paper, as exact tests on the counts:")
for name, a, b, n in (("Bridge: OpenVLA vs RT-2-X (120 vs 86 of 170)", 120, 86, 170), ("Bridge: OpenVLA vs Octo (120 vs 34 of 170)", 120, 34, 170),
                      ("Google: OpenVLA vs RT-2-X (51 vs 47 of 60)", 51, 47, 60), ("Franka: OpenVLA vs Diffusion Policy (84 vs 59 of 129)", 84, 59, 129),
                      ("Table 9: OpenVLA vs Bridge-only (61 vs 36 of 80)", 61, 36, 80), ("Table 9: Bridge-only vs no DinoV2 (36 vs 32 of 80)", 36, 32, 80),
                      ("Table 1: LoRA vs full finetuning (22 vs 23 of 33)", 22, 23, 33)):
    print(f"  {name:55s} {100 * a / n:5.1f}% vs {100 * b / n:5.1f}%: p = {fisher(a, n, b, n):.4f}")
print()
# one task, one method: what 10 trials can tell you
print("a single Bridge task with 10 trials: 95% interval of the true success rate for typical counts (Wilson)")
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d; h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return 100 * (c - h), 100 * (c + h)
for k in (2, 5, 8):
    lo, hi = wilson(k, 10); print(f"  {k}/10 -> {lo:4.0f}% to {hi:3.0f}%")

Output:

trials | where in the paper                                       | StdErr at 50% | smallest gap an exact test calls different (baseline at 50%)
     5 | Google robot, per task; Franka OOD rows                  |      22.4 pts | none
    10 | Bridge, per task; Franka in-distribution rows            |      15.8 pts | 50 points
    33 | Table 1, per strategy                                    |       8.7 pts | 27 points
    60 | Google robot, per method; Table 10                       |       6.5 pts | 20 points
    80 | Tables 2, 5, 9, 11                                       |       5.6 pts | 18 points
    99 | Franka-Tabletop, per method                              |       5.0 pts | 15 points
   129 | all Franka rollouts                                      |       4.4 pts | 13 points
   170 | Bridge, per method                                       |       3.8 pts | 11 points

headline gaps in the paper, as exact tests on the counts:
  Bridge: OpenVLA vs RT-2-X (120 vs 86 of 170)             70.6% vs  50.6%: p = 0.0002
  Bridge: OpenVLA vs Octo (120 vs 34 of 170)               70.6% vs  20.0%: p = 0.0000
  Google: OpenVLA vs RT-2-X (51 vs 47 of 60)               85.0% vs  78.3%: p = 0.4798
  Franka: OpenVLA vs Diffusion Policy (84 vs 59 of 129)    65.1% vs  45.7%: p = 0.0026
  Table 9: OpenVLA vs Bridge-only (61 vs 36 of 80)         76.2% vs  45.0%: p = 0.0001
  Table 9: Bridge-only vs no DinoV2 (36 vs 32 of 80)       45.0% vs  40.0%: p = 0.6315
  Table 1: LoRA vs full finetuning (22 vs 23 of 33)        66.7% vs  69.7%: p = 1.0000

a single Bridge task with 10 trials: 95% interval of the true success rate for typical counts (Wilson)
  2/10 ->    6% to  51%
  5/10 ->   24% to  76%
  8/10 ->   49% to  94%

Our calculation, assuming independent trials; each experiment is a single run.

Evidence in the paper

§5.1 · §5.2 · Appendix B

Check your understanding: Why are all comparisons run as A/B tests on the same initial states?

With 10 trials per task, differences between initial states could swamp differences between policies; using the same starting scenes for every method removes that source of variation, though not the randomness within a method.

Direct evaluations: OpenVLA against RT-2-X, Octo and RT-1-X (Tables 4 and 6)

How it works

On the Bridge robot the averages are 70.6% for OpenVLA, 50.6% for RT-2-X, 20.0% for Octo and 18.5% for RT-1-X over 170 rollouts. On the Google robot they are 85.0%, 78.3%, 26.7% and 33.3% over 60. OpenVLA is clearly ahead on the visual and physical tasks, about level on the motion and language-grounding tasks, and slightly behind on the semantic ones; on the Google robot the two are level. The authors credit the larger and cleaner dataset and the fused encoder, and note that RT-2-X also had a co-training recipe designed to keep web knowledge.

Technical details

Tables 4 and 6, mean success and standard error in percent, as printed.
Method	Bridge (170 rollouts)	Google robot (60 rollouts)
RT-1-X	18.5 ± 2.7	33.3 ± 6.1
Octo	20.0 ± 2.6	26.7 ± 5.8
RT-2-X	50.6 ± 3.5	78.3 ± 5.4
OpenVLA	70.6 ± 3.2	85.0 ± 4.6

Our calculation The per-task counts reproduce all eight means. Pooled over the 230 rollouts, OpenVLA is 74.35% and RT-2-X 57.83%, a difference of 16.5 points: that is the abstract’s “16.5% over 29 tasks”. It is 20.0 points on Bridge (120 against 86 successes of 170, exact p of 0.0002) and 6.7 on the Google robot (51 against 47 of 60, p of 0.480). The printed standard errors for Bridge are smaller than the binomial values (2.7 against 3.0 for RT-1-X, 3.2 against 3.5 for OpenVLA); the paper does not give the formula.

Run the example: means, standard errors and the 16.5% claim

Our calculation

What each part does

Type in the per-task counts and recompute the eight means.
Pool the two robots and compare with the abstract’s 16.5%.
Run an exact test per robot and add up the counts per category.

import math
# Tables 4 and 6: recompute the printed means from the per-task success counts (half successes are partial credit).
bridge = {  # task: (RT-1-X, Octo, RT-2-X, OpenVLA) successes out of 10
 "Eggplant into Pot (easy)": (1, 5, 7, 10), "Eggplant into Pot": (0, 1, 5, 10), "Cup from Counter into Sink": (1, 1, 0, 7), "Eggplant into Pot (clutter)": (1, 3.5, 6, 7.5),
 "Yellow Corn on Pink Plate": (1, 4, 8, 9), "Lift Eggplant": (3, 0.5, 6.5, 7.5), "Carrot on Plate (height change)": (2, 1, 4.5, 4.5), "Carrot on Plate": (1, 0, 1, 8),
 "Flip Pot Upright": (2, 6, 5, 8), "Lift AAA Battery": (0, 0, 2, 7), "Skull into Drying Rack": (1, 0, 5, 5), "Lift White Tape": (3, 0, 0, 1),
 "Purple Grapes out of Pot": (6, 0, 5, 4), "Stack Blue Cup on Pink Cup": (0.5, 0, 5.5, 4.5), "{Eggplant, Red Bottle} into Pot": (2.5, 4, 8.5, 7.5),
 "Lift {Cheese, Chili}": (1.5, 2.5, 8.5, 10), "{Blue Cup, Pink Cup} on Plate": (5, 5.5, 8.5, 9.5)}
names = ["RT-1-X", "Octo", "RT-2-X", "OpenVLA"]
printed_b = [18.5, 20.0, 50.6, 70.6]; printed_se = [2.7, 2.6, 3.5, 3.2]
google = (  # (RT-1-X, Octo, RT-2-X, OpenVLA) successes out of 5; first five in-distribution, last seven out-of-distribution
 (5, 1, 5, 5), (3, 3, 3, 5), (0, 3, 4, 5), (0, 0, 4, 4), (0, 4, 2, 3), (1, 2, 5, 5), (3, 0, 5, 4), (5, 3, 5, 5), (1, 0, 5, 5), (0, 0, 4, 4), (0, 0, 2, 4), (2, 0, 3, 2))
printed_g = [33.3, 26.7, 78.3, 85.0]
print("Bridge WidowX: 17 tasks x 10 trials = 170 rollouts")
for j, n in enumerate(names):
    s = sum(v[j] for v in bridge.values()); p = s / 170
    print(f"  {n:8s} successes {s:6.1f}/170  mean {100 * p:5.2f}%  printed {printed_b[j]}%  binomial StdErr {100 * math.sqrt(p * (1 - p) / 170):.1f} vs printed {printed_se[j]}")
print("Google robot: 12 tasks x 5 trials = 60 rollouts")
for j, n in enumerate(names):
    s = sum(v[j] for v in google); p = s / 60
    print(f"  {n:8s} successes {s:4.0f}/60    mean {100 * p:5.2f}%  printed {printed_g[j]}%  binomial StdErr {100 * math.sqrt(p * (1 - p) / 60):.1f}")
print()
# the abstract's "16.5% over 29 tasks": pooled OpenVLA minus RT-2-X
ov, rt2 = [sum(v[j] for v in bridge.values()) + sum(v[j] for v in google) for j in (3, 2)]
print(f"pooled over 230 rollouts: OpenVLA {100 * ov / 230:.2f}%, RT-2-X {100 * rt2 / 230:.2f}%, difference {100 * (ov - rt2) / 230:.1f} points (abstract: 16.5)")
print(f"unweighted mean of the two suite averages: {(70.6 + 85.0) / 2 - (50.6 + 78.3) / 2:.1f} points; Bridge alone {70.6 - 50.6:.1f}, Google alone {85.0 - 78.3:.1f}")
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
bo, br = round(sum(v[3] for v in bridge.values())), round(sum(v[2] for v in bridge.values()))
go, gr = sum(v[3] for v in google), sum(v[2] for v in google)
print(f"exact test (half successes rounded): Bridge OpenVLA {bo}/170 vs RT-2-X {br}/170 p = {fisher(bo, 170, br, 170):.5f}; Google {go}/60 vs {gr}/60 p = {fisher(go, 60, gr, 60):.3f}")
print("by category on Bridge (successes/trials), OpenVLA vs RT-2-X:")
cats = {"visual (5 tasks)": range(0, 5), "motion (2)": range(5, 7), "physical (3)": range(7, 10), "semantic (4)": range(10, 14), "language grounding (3)": range(14, 17)}
rows = list(bridge.values())
for c, idx in cats.items():
    n = 10 * len(idx)
    print(f"  {c:24s} OpenVLA {sum(rows[i][3] for i in idx):5.1f}/{n}  RT-2-X {sum(rows[i][2] for i in idx):5.1f}/{n}")

Output:

Bridge WidowX: 17 tasks x 10 trials = 170 rollouts
  RT-1-X   successes   31.5/170  mean 18.53%  printed 18.5%  binomial StdErr 3.0 vs printed 2.7
  Octo     successes   34.0/170  mean 20.00%  printed 20.0%  binomial StdErr 3.1 vs printed 2.6
  RT-2-X   successes   86.0/170  mean 50.59%  printed 50.6%  binomial StdErr 3.8 vs printed 3.5
  OpenVLA  successes  120.0/170  mean 70.59%  printed 70.6%  binomial StdErr 3.5 vs printed 3.2
Google robot: 12 tasks x 5 trials = 60 rollouts
  RT-1-X   successes   20/60    mean 33.33%  printed 33.3%  binomial StdErr 6.1
  Octo     successes   16/60    mean 26.67%  printed 26.7%  binomial StdErr 5.7
  RT-2-X   successes   47/60    mean 78.33%  printed 78.3%  binomial StdErr 5.3
  OpenVLA  successes   51/60    mean 85.00%  printed 85.0%  binomial StdErr 4.6

pooled over 230 rollouts: OpenVLA 74.35%, RT-2-X 57.83%, difference 16.5 points (abstract: 16.5)
unweighted mean of the two suite averages: 13.3 points; Bridge alone 20.0, Google alone 6.7
exact test (half successes rounded): Bridge OpenVLA 120/170 vs RT-2-X 86/170 p = 0.00024; Google 51/60 vs 47/60 p = 0.480
by category on Bridge (successes/trials), OpenVLA vs RT-2-X:
  visual (5 tasks)         OpenVLA  43.5/50  RT-2-X  26.0/50
  motion (2)               OpenVLA  12.0/20  RT-2-X  11.0/20
  physical (3)             OpenVLA  23.0/30  RT-2-X   8.0/30
  semantic (4)             OpenVLA  14.5/40  RT-2-X  15.5/40
  language grounding (3)   OpenVLA  27.0/30  RT-2-X  25.5/30

Arithmetic on the printed tables; half successes are partial credit and are rounded for the tests.

Evidence in the paper

§5.1 · Figures 3 and 4 · Appendix B.1 (Table 4) · Appendix B.2 (Table 6)

Check your understanding: Is “OpenVLA beats RT-2-X” the same claim on both robots?

No. On the Bridge robot the gap is large and clearly beyond chance. On the Google robot it is 4 more successes out of 60, which an exact test cannot tell from chance, and the paper itself calls the two comparable there.

Finetuning to a new robot (Table 7)

How it works

Seven Franka tasks with 10 to 150 demonstrations each. OpenVLA averages 67.2% on the six tabletop tasks (99 rollouts) and 58.3% on the Franka-DROID task (30 rollouts); Diffusion Policy gets 48.5% and 35.0%, a finetuned Octo 43.4% and 38.3%, and an OpenVLA trained from the base vision-language model without the robot pretraining 43.4% and 21.7%. Diffusion Policy is best on the narrow tasks and the pretrained generalists are better on tasks with several objects and instructions. The authors say OpenVLA is the only method with at least 50% on every task.

Technical details

Task-group averages are our calculation from Table 7 (rounded counts), in percent.
Task group	Diffusion Policy	Octo	OpenVLA
Narrow single-instruction (3 tasks, 45 rollouts)	81.1	20.0	68.9
Several objects and instructions (3 tasks, 54 rollouts)	21.3	63.0	65.7

Our calculation OpenVLA is level with Octo on the multi-instruction tasks (35 against 34 successes of 54, p of 1.0) and far ahead of Octo on the narrow ones (31 against 9 of 45). Its lead over Diffusion Policy comes entirely from the multi-instruction tasks; on the narrow ones Diffusion Policy is ahead, although the gap is within chance (36 against 31 of 45, p of 0.334). The abstract’s “20.4%” gain over Diffusion Policy is not reproduced exactly by any aggregation we tried: 19.8 points pooled over the 129 rollouts, 21.0 as the mean of the two setup averages.

Run the example: per-task numbers and the 20.4% claim

Our calculation

What each part does

Type in Table 7 and recompute the averages with the trial counts as weights.
Pool in-distribution and out-of-distribution rows to find each task’s rate and test the “at least 50%” claim.
Try several ways of aggregating the gain over Diffusion Policy and test the groups.

import math
# Table 7: data-efficient adaptation. Columns: Diffusion Policy, Diffusion Policy (matched), Octo, OpenVLA (scratch), OpenVLA. Success in percent.
methods = ["Diffusion Policy", "DP (matched)", "Octo", "OpenVLA (scratch)", "OpenVLA"]
tab = [  # (task, setting, trials, five success rates)
 ("Put Carrot in Bowl", "in", 10, (90.0, 80.0, 40.0, 70.0, 70.0)), ("Put Carrot in Bowl", "ood", 5, (20.0, 0.0, 20.0, 0.0, 40.0)),
 ("Pour Corn into Pot", "in", 10, (100.0, 90.0, 0.0, 10.0, 50.0)), ("Pour Corn into Pot", "ood", 5, (80.0, 60.0, 0.0, 20.0, 60.0)),
 ("Flip Pot Upright", "in", 10, (100.0, 85.0, 40.0, 85.0, 100.0)), ("Flip Pot Upright", "ood", 5, (50.0, 20.0, 0.0, 40.0, 80.0)),
 ("Move onto Plate", "in", 12, (25.0, 25.0, 41.7, 8.3, 75.0)), ("Move onto Plate", "ood", 6, (8.3, 33.3, 8.3, 33.3, 58.3)),
 ("Knock Over", "in", 12, (33.3, 25.0, 83.3, 75.0, 75.0)), ("Knock Over", "ood", 6, (16.7, 16.7, 33.3, 58.3, 83.3)),
 ("Cover with Towel", "in", 12, (16.7, 20.8, 91.7, 41.7, 50.0)), ("Cover with Towel", "ood", 6, (16.7, 33.3, 91.7, 50.0, 50.0))]
droid = [("Wipe Table", "in", 18, (50.0, 27.8, 52.8, 25.0, 55.6)), ("Wipe Table", "ood", 12, (12.5, 25.0, 16.7, 16.7, 62.5))]
printed_t = [48.5, 43.4, 43.4, 43.4, 67.2]; printed_d = [35.0, 26.7, 38.3, 21.7, 58.3]
N = sum(r[2] for r in tab); Nd = sum(r[2] for r in droid)
print(f"Franka-Tabletop: {N} rollouts; Franka-DROID: {Nd} rollouts (each scored 0, 1 or 2 points, 60 points in total)")
mt, md = [], []
for j, m in enumerate(methods):
    a = sum(r[2] * r[3][j] for r in tab) / N; b = sum(r[2] * r[3][j] for r in droid) / Nd
    se = 100 * math.sqrt(a / 100 * (1 - a / 100) / N)
    mt.append(a); md.append(b)
    print(f"  {m:18s} Tabletop {a:5.1f}% (printed {printed_t[j]}, binomial StdErr {se:.1f})   DROID {b:5.1f}% (printed {printed_d[j]})")
print()
print("per task, in-distribution and out-of-distribution pooled (the numbers behind Figure 5):")
tasks = []
for t, *_ in tab + droid:
    if t not in tasks: tasks.append(t)
pooled = {}
for t in tasks:
    rows = [r for r in tab + droid if r[0] == t]; n = sum(r[2] for r in rows)
    pooled[t] = [sum(r[2] * r[3][j] for r in rows) / n for j in range(5)]
    print(f"  {t:24s} " + "  ".join(f"{v:5.1f}" for v in pooled[t]))
print("  columns:", ", ".join(methods))
print()
print("'OpenVLA is the only approach with at least 50% on every task' (pooled per task):")
for j, m in enumerate(methods):
    print(f"  {m:18s} lowest task {min(pooled[t][j] for t in tasks):5.1f}%  tasks below 50%: {sum(pooled[t][j] < 50 for t in tasks)}")
print()
# the abstract: "outperform Diffusion Policy by 20.4%". Several ways of aggregating
w = (mt[4] * N + md[4] * Nd) / (N + Nd) - (mt[0] * N + md[0] * Nd) / (N + Nd)
print("OpenVLA minus Diffusion Policy, by aggregation (abstract says 20.4):")
print(f"  pooled over {N + Nd} rollouts      {w:5.1f} points")
print(f"  mean of the two setup averages    {(mt[4] + md[4]) / 2 - (mt[0] + md[0]) / 2:5.1f} points")
print(f"  mean of the seven task averages   {sum(pooled[t][4] - pooled[t][0] for t in tasks) / 7:5.1f} points")
print(f"  Tabletop alone {mt[4] - mt[0]:.1f}; DROID alone {md[4] - md[0]:.1f}")
single, multi = tasks[:3], tasks[3:6]
print(f"  narrow single-instruction tasks: OpenVLA {sum(pooled[t][4] for t in single) / 3:.1f} vs Diffusion Policy {sum(pooled[t][0] for t in single) / 3:.1f}")
print(f"  diverse multi-instruction tasks: OpenVLA {sum(pooled[t][4] for t in multi) / 3:.1f} vs Diffusion Policy {sum(pooled[t][0] for t in multi) / 3:.1f}")
print()
def fisher(a, n1, b, n2):
    m, T = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(T - m, n1 - x) / math.comb(T, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
def counts(rows, j): return round(sum(r[2] * r[3][j] / 100 for r in rows)), sum(r[2] for r in rows)
print("exact tests on rounded success counts (half successes and the DROID 2-point scores are approximated as rates):")
for label, rows in (("all 129 rollouts", tab + droid), ("narrow single-instruction tasks", tab[:6]), ("diverse multi-instruction tasks", tab[6:12])):
    o, n = counts(rows, 4); d, _ = counts(rows, 0); t, _ = counts(rows, 2)
    print(f"  {label:32s} OpenVLA {o}/{n} vs Diffusion Policy {d}/{n} p = {fisher(o, n, d, n):.4f};  vs Octo {t}/{n} p = {fisher(o, n, t, n):.4f}")

Output:

Franka-Tabletop: 99 rollouts; Franka-DROID: 30 rollouts (each scored 0, 1 or 2 points, 60 points in total)
  Diffusion Policy   Tabletop  48.5% (printed 48.5, binomial StdErr 5.0)   DROID  35.0% (printed 35.0)
  DP (matched)       Tabletop  43.4% (printed 43.4, binomial StdErr 5.0)   DROID  26.7% (printed 26.7)
  Octo               Tabletop  43.4% (printed 43.4, binomial StdErr 5.0)   DROID  38.4% (printed 38.3)
  OpenVLA (scratch)  Tabletop  43.4% (printed 43.4, binomial StdErr 5.0)   DROID  21.7% (printed 21.7)
  OpenVLA            Tabletop  67.2% (printed 67.2, binomial StdErr 4.7)   DROID  58.4% (printed 58.3)

per task, in-distribution and out-of-distribution pooled (the numbers behind Figure 5):
  Put Carrot in Bowl        66.7   53.3   33.3   46.7   60.0
  Pour Corn into Pot        93.3   80.0    0.0   13.3   53.3
  Flip Pot Upright          83.3   63.3   26.7   70.0   93.3
  Move onto Plate           19.4   27.8   30.6   16.6   69.4
  Knock Over                27.8   22.2   66.6   69.4   77.8
  Cover with Towel          16.7   25.0   91.7   44.5   50.0
  Wipe Table                35.0   26.7   38.4   21.7   58.4
  columns: Diffusion Policy, DP (matched), Octo, OpenVLA (scratch), OpenVLA

'OpenVLA is the only approach with at least 50% on every task' (pooled per task):
  Diffusion Policy   lowest task  16.7%  tasks below 50%: 4
  DP (matched)       lowest task  22.2%  tasks below 50%: 4
  Octo               lowest task   0.0%  tasks below 50%: 5
  OpenVLA (scratch)  lowest task  13.3%  tasks below 50%: 5
  OpenVLA            lowest task  50.0%  tasks below 50%: 0

OpenVLA minus Diffusion Policy, by aggregation (abstract says 20.4):
  pooled over 129 rollouts       19.8 points
  mean of the two setup averages     21.0 points
  mean of the seven task averages    17.1 points
  Tabletop alone 18.7; DROID alone 23.4
  narrow single-instruction tasks: OpenVLA 68.9 vs Diffusion Policy 81.1
  diverse multi-instruction tasks: OpenVLA 65.7 vs Diffusion Policy 21.3

exact tests on rounded success counts (half successes and the DROID 2-point scores are approximated as rates):
  all 129 rollouts                 OpenVLA 84/129 vs Diffusion Policy 59/129 p = 0.0026;  vs Octo 55/129 p = 0.0004
  narrow single-instruction tasks  OpenVLA 31/45 vs Diffusion Policy 36/45 p = 0.3339;  vs Octo 9/45 p = 0.0000
  diverse multi-instruction tasks  OpenVLA 35/54 vs Diffusion Policy 12/54 p = 0.0000;  vs Octo 34/54 p = 1.0000

Arithmetic on the printed table; the 2-point scores of the DROID task are treated as rates.

Evidence in the paper

§5.2 · Figure 5 · Appendix B.3 (Table 7)

Check your understanding: For which kind of task does the paper recommend OpenVLA?

Tasks with several objects and language instructions; for narrow, highly dexterous tasks it concedes that Diffusion Policy gives smoother and more precise motion, and suggests action chunking as a possible remedy.

Cheaper finetuning: LoRA (Tables 1 and 8)

How it works

Low-rank adaptation (LoRA) leaves the original weights untouched and trains small extra matrices on every linear layer. With rank 32 it trains 97.6 million parameters, 1.4% of the full 7,188.1 million, and scores 68.2% on 33 rollouts of two Franka tasks, against 69.7% for full finetuning. Training only the last layer (30.3%) or freezing the vision encoder (47.0%) is worse; “sandwich” finetuning of the vision encoder, embeddings and last layer gets 62.1%. The authors report that LoRA takes 10 to 15 hours on one A100 card, “an 8x reduction” from full finetuning on 8 cards.

Technical details

Our calculation The printed means reproduce from the per-task rates. The 33 rollouts cannot separate LoRA from full finetuning (22 against 23 successes, p of 1.0) or from sandwich finetuning, and full finetuning against a frozen vision encoder has p of 0.080; only the last-layer-only result is clearly worse (p of 0.003). The “8x” holds at one end of the range: 8 cards for 15 hours is 120 card-hours, 12 times a 10-hour LoRA run and 8 times a 15-hour one, but 8 cards for 5 hours is 4 times a 10-hour run.

Run the example: LoRA, trainable parameters and compute

Our calculation

What each part does

Recompute each strategy’s mean from the four task rows and the share of trainable weights.
Test full finetuning against each alternative.
Check the “8x” claim for the stated hours.

import math
# Tables 1 and 8: parameter-efficient finetuning. 33 rollouts per strategy: 10 + 5 for Put Carrot in Bowl, 12 + 6 for Move onto Plate. Success in percent.
trials = [10, 5, 12, 6]
strategies = {  # (four task rates), printed mean, trainable parameters in millions, memory in GB at batch 16
 "Full finetuning": ((90.0, 40.0, 79.2, 41.7), 69.7, 7188.1, 163.3), "Last layer only": ((40.0, 0.0, 33.3, 33.3), 30.3, 465.1, 51.4),
 "Frozen vision": ((40.0, 40.0, 50.0, 58.3), 47.0, 6760.4, 156.2), "Sandwich": ((90.0, 0.0, 75.0, 41.7), 62.1, 914.2, 64.0),
 "LoRA rank 32": ((60.0, 60.0, 75.0, 75.0), 68.2, 97.6, 59.7), "LoRA rank 64": ((90.0, 40.0, 62.5, 66.7), 68.2, 195.2, 60.5)}
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
print(f"{'strategy':16s} {'mean':>6s} {'printed':>7s} {'StdErr':>6s} {'trained':>9s} {'of full':>7s} {'GB':>6s}")
full = strategies["Full finetuning"]
means = {}
for k, (r, printed, par, gb) in strategies.items():
    m = sum(t * x for t, x in zip(trials, r)) / 33
    means[k] = m
    print(f"{k:16s} {m:6.1f} {printed:7.1f} {100 * math.sqrt(m / 100 * (1 - m / 100) / 33):6.1f} {par:8.1f}M {100 * par / full[2]:6.2f}% {gb:6.1f}")
print()
cnt = lambda k: round(means[k] * 33 / 100)
for k in ("Sandwich", "LoRA rank 32", "Frozen vision", "Last layer only"):
    print(f"Full finetuning {cnt('Full finetuning')}/33 vs {k} {cnt(k)}/33: exact p = {fisher(cnt('Full finetuning'), 33, cnt(k), 33):.3f}")
print(f"LoRA rank 32 {cnt('LoRA rank 32')}/33 vs Sandwich {cnt('Sandwich')}/33: exact p = {fisher(cnt('LoRA rank 32'), 33, cnt('Sandwich'), 33):.3f}")
print()
print("the two task types separately (percent):")
for k, (r, *_ ) in strategies.items():
    narrow = (10 * r[0] + 5 * r[1]) / 15; multi = (12 * r[2] + 6 * r[3]) / 18
    print(f"  {k:16s} Put Carrot in Bowl {narrow:5.1f}   Move onto Plate {multi:5.1f}")
print()
# compute claims
print("compute: 'within 10-15 hours on one A100, an 8x reduction' against 8 A100s for 5-15 hours")
for fh in (5, 15):
    print(f"  full finetuning {fh} h x 8 GPUs = {8 * fh} A100-hours; LoRA 10 h -> {8 * fh / 10:.1f}x, 15 h -> {8 * fh / 15:.1f}x")
print(f"  pretraining: 64 GPUs x 14 days x 24 h = {64 * 14 * 24:,} A100-hours (paper: 21,500)")
print(f"  one full finetune at 40-120 A100-hours is {100 * 40 / 21504:.2f}% to {100 * 120 / 21504:.2f}% of pretraining")
print()
print("memory: trained parameters x bytes, a rough floor (the printed memory also holds activations, optimiser state and data)")
p = 7188.1e6
print(f"  weights in bfloat16 (2 bytes): {p * 2 / 1e9:.1f} GB; printed inference memory 16.8 GB (Table 2) and 15 GB (Section 3.5)")
print(f"  full finetuning with Adam (weights 2 + gradients 2 + two moments 4 + master copy 4 bytes, an assumption): {p * 12 / 1e9:.0f} GB before activations; printed 163.3 GB over 2 GPUs")

Output:

strategy           mean printed StdErr   trained of full     GB
Full finetuning    69.7    69.7    8.0   7188.1M 100.00%  163.3
Last layer only    30.3    30.3    8.0    465.1M   6.47%   51.4
Frozen vision      47.0    47.0    8.7   6760.4M  94.05%  156.2
Sandwich           62.1    62.1    8.4    914.2M  12.72%   64.0
LoRA rank 32       68.2    68.2    8.1     97.6M   1.36%   59.7
LoRA rank 64       68.2    68.2    8.1    195.2M   2.72%   60.5

Full finetuning 23/33 vs Sandwich 21/33: exact p = 0.794
Full finetuning 23/33 vs LoRA rank 32 22/33: exact p = 1.000
Full finetuning 23/33 vs Frozen vision 15/33: exact p = 0.080
Full finetuning 23/33 vs Last layer only 10/33: exact p = 0.003
LoRA rank 32 22/33 vs Sandwich 21/33: exact p = 1.000

the two task types separately (percent):
  Full finetuning  Put Carrot in Bowl  73.3   Move onto Plate  66.7
  Last layer only  Put Carrot in Bowl  26.7   Move onto Plate  33.3
  Frozen vision    Put Carrot in Bowl  40.0   Move onto Plate  52.8
  Sandwich         Put Carrot in Bowl  60.0   Move onto Plate  63.9
  LoRA rank 32     Put Carrot in Bowl  60.0   Move onto Plate  75.0
  LoRA rank 64     Put Carrot in Bowl  73.3   Move onto Plate  63.9

compute: 'within 10-15 hours on one A100, an 8x reduction' against 8 A100s for 5-15 hours
  full finetuning 5 h x 8 GPUs = 40 A100-hours; LoRA 10 h -> 4.0x, 15 h -> 2.7x
  full finetuning 15 h x 8 GPUs = 120 A100-hours; LoRA 10 h -> 12.0x, 15 h -> 8.0x
  pretraining: 64 GPUs x 14 days x 24 h = 21,504 A100-hours (paper: 21,500)
  one full finetune at 40-120 A100-hours is 0.19% to 0.56% of pretraining

memory: trained parameters x bytes, a rough floor (the printed memory also holds activations, optimiser state and data)
  weights in bfloat16 (2 bytes): 14.4 GB; printed inference memory 16.8 GB (Table 2) and 15 GB (Section 3.5)
  full finetuning with Adam (weights 2 + gradients 2 + two moments 4 + master copy 4 bytes, an assumption): 86 GB before activations; printed 163.3 GB over 2 GPUs

Arithmetic on Tables 1 and 8; the optimiser-memory estimate is our assumption.

Evidence in the paper

§5.3 · Table 1 · Appendix B.3 (Table 8) · Footnote 4

Check your understanding: What does “LoRA matches full finetuning” rest on?

A 1.5-point difference in 33 rollouts of two tasks, on a smaller variant of the model, with one run per row. It shows that LoRA did not lose a lot; it does not show it loses nothing.

Cheaper serving: 8-bit and 4-bit (Tables 2, 5 and 11)

How it works

Loading the weights in bfloat16, a 16-bit format, needs 16.8 GB. 8-bit uses 10.2 GB and 4-bit 7.0 GB. On eight Bridge tasks (80 rollouts) the success rates are 71.3% (bfloat16), 58.1% (8-bit) and 71.9% (4-bit). The authors explain the drop for 8-bit by slower inference: on their A5000 card it runs at 1.2 Hz, while the controller assumes 5 Hz, and 4-bit runs at 3 Hz. Repeating the test with blocking control, where the robot waits for each prediction, gives 70.0%, 74.4% and 68.8%, which supports the explanation.

Technical details

Our calculation The totals reproduce (57, 46.5 and 57.5 successes of 80). The 8-bit drop in the non-blocking test is 10.5 successes, with p of 0.098, so even that result is only suggestive. The same bfloat16 model scores 57 of 80 in both control modes, yet single tasks move by up to 6 of 10 trials (Purple Grapes: 8 then 2; Lift AAA Battery: 7 then 3), a direct view of run-to-run noise.

Run the example: quantisation, speed and noise

Our calculation

What each part does

Recompute the means for both control modes.
Test bfloat16 against 8-bit and 4-bit.
Compare the same model across control modes, per task.

import math
# Tables 5 and 11 (and Table 2): eight Bridge tasks x 10 trials = 80 rollouts. Half successes are partial credit.
tasks = ["Eggplant in Pot (easy)", "Eggplant in Pot", "Cup into Sink", "Lift Eggplant", "Carrot on Plate", "Lift AAA Battery", "Purple Grapes", "{Eggplant, Bottle} in Pot"]
tab5 = {"bfloat16": (9, 7, 5, 6, 6, 7, 8, 9), "int8": (7, 7, 3, 4, 5, 5, 8, 7.5), "int4": (9, 7, 7, 7.5, 7, 3, 9, 8)}      # non-blocking control
tab11 = {"bfloat16": (10, 9, 5, 8, 10, 3, 2, 9), "int8": (10, 10, 5, 7, 10, 6, 2, 9.5), "int4": (10, 10, 3, 7.5, 10, 4, 2, 8.5)}  # blocking control
printed5 = {"bfloat16": 71.3, "int8": 58.1, "int4": 71.9}; printed11 = {"bfloat16": 70.0, "int8": 74.4, "int4": 68.8}
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
for name, tab, printed in (("Table 5 (non-blocking control, speed matters)", tab5, printed5), ("Table 11 (blocking control, speed removed)", tab11, printed11)):
    print(name)
    for k, v in tab.items():
        s = sum(v); p = s / 80
        print(f"  {k:9s} {s:5.1f}/80  mean {100 * p:5.1f}% printed {printed[k]}%  binomial StdErr {100 * math.sqrt(p * (1 - p) / 80):.1f}")
    b, i8, i4 = (round(sum(tab[k])) for k in ("bfloat16", "int8", "int4"))
    print(f"  exact tests on rounded counts: bfloat16 vs int8 p = {fisher(b, 80, i8, 80):.3f}; bfloat16 vs int4 p = {fisher(b, 80, i4, 80):.3f}")
print()
print("the same bfloat16 model, same tasks, only the control mode changes:")
d = [b - a for a, b in zip(tab5["bfloat16"], tab11["bfloat16"])]
print("  per-task change (blocking minus non-blocking):", d)
print(f"  Purple Grapes alone: 8/10 in Table 5 and 2/10 in Table 11; Lift AAA Battery: 7/10 and 3/10 for bfloat16")
print(f"  total {sum(tab5['bfloat16']):.0f} vs {sum(tab11['bfloat16']):.0f} of 80; exact p = {fisher(round(sum(tab5['bfloat16'])), 80, round(sum(tab11['bfloat16'])), 80):.3f}")
print()
print("control rate against the model's speed (Section 5.4):")
for name, hz in (("controller in the BridgeData V2 tasks", 5.0), ("int8 on the A5000", 1.2), ("int4 on the A5000", 3.0)):
    print(f"  {name:38s} {hz:4.1f} Hz -> one action every {1000 / hz:5.0f} ms")
print("  If actions are only produced at 1.2 Hz while the robot expects 5 Hz, each prediction covers about", round(5.0 / 1.2, 1), "control steps.")
print()
print("memory: bfloat16 16.8 GB, int8 10.2 GB, int4 7.0 GB ->", f"{10.2 / 16.8:.0%} and {7.0 / 16.8:.0%} of bfloat16")
print("weights alone at 7.19 billion parameters (the 7,188.1 million of Table 1):", ", ".join(f"{n} {7.1881 * b:.1f} GB" for n, b in (("bfloat16", 2), ("int8", 1), ("int4", 0.5))))

Output:

Table 5 (non-blocking control, speed matters)
  bfloat16   57.0/80  mean  71.2% printed 71.3%  binomial StdErr 5.1
  int8       46.5/80  mean  58.1% printed 58.1%  binomial StdErr 5.5
  int4       57.5/80  mean  71.9% printed 71.9%  binomial StdErr 5.0
  exact tests on rounded counts: bfloat16 vs int8 p = 0.098; bfloat16 vs int4 p = 1.000
Table 11 (blocking control, speed removed)
  bfloat16   56.0/80  mean  70.0% printed 70.0%  binomial StdErr 5.1
  int8       59.5/80  mean  74.4% printed 74.4%  binomial StdErr 4.9
  int4       55.0/80  mean  68.8% printed 68.8%  binomial StdErr 5.2
  exact tests on rounded counts: bfloat16 vs int8 p = 0.596; bfloat16 vs int4 p = 1.000

the same bfloat16 model, same tasks, only the control mode changes:
  per-task change (blocking minus non-blocking): [1, 2, 0, 2, 4, -4, -6, 0]
  Purple Grapes alone: 8/10 in Table 5 and 2/10 in Table 11; Lift AAA Battery: 7/10 and 3/10 for bfloat16
  total 57 vs 56 of 80; exact p = 1.000

control rate against the model's speed (Section 5.4):
  controller in the BridgeData V2 tasks   5.0 Hz -> one action every   200 ms
  int8 on the A5000                       1.2 Hz -> one action every   833 ms
  int4 on the A5000                       3.0 Hz -> one action every   333 ms
  If actions are only produced at 1.2 Hz while the robot expects 5 Hz, each prediction covers about 4.2 control steps.

memory: bfloat16 16.8 GB, int8 10.2 GB, int4 7.0 GB -> 61% and 42% of bfloat16
weights alone at 7.19 billion parameters (the 7,188.1 million of Table 1): bfloat16 14.4 GB, int8 7.2 GB, int4 3.6 GB

Arithmetic on Tables 5 and 11; half successes are rounded for the tests.

Evidence in the paper

§5.4 · Table 2 · Figure 6 · Appendix D.4 (Tables 5 and 11)

Check your understanding: Why does the authors’ explanation of the 8-bit result matter?

It separates the effect of precision on the model’s answers from the effect of latency on the robot. Under blocking control the 8-bit model is as good as the others, so lower precision is not the cause.

What matters: Open X-Embodiment data, the second encoder, finetuning the encoder (Tables 9 and 10)

How it works

On eight Bridge tasks (80 rollouts) the full OpenVLA scores 76.3%; the same architecture trained on Bridge data alone (OpenVLA-Bridge) 45.6%; and the Bridge-only model with DINOv2 removed (OpenVLA-Bridge-SigLIP) 40.6%. The data effect is 30 points and the encoder effect 5. Separately, two vision-language backbones finetuned on Bridge with a frozen or finetuned image encoder give 46.7% against 80.0% over 60 rollouts.

Technical details

Table 9, as printed, with the successes added by us.
Variant	Success (%)	Successes of 80
OpenVLA (full mixture, fused encoder)	76.3	61.0
OpenVLA-Bridge (Bridge data only)	45.6	36.5
OpenVLA-Bridge-SigLIP (no DINOv2)	40.6	32.5

Our calculation The 30.6-point data effect is far beyond chance (exact p of 0.0001); the 5-point DINOv2 effect is not (p of 0.632). DINOv2 helps on 3 tasks, ties on 3 and hurts on 2. Pooling Table 10 over its 60 rollouts reproduces 46.7% and 80.0% (p of 0.0003).

Run the example: what matters in the recipe

Our calculation

What each part does

Recompute the three means and the per-task differences.
Test the data effect and the encoder effect.
Pool Table 10 over its blocks.

import math
# Tables 9 and 10: what matters in OpenVLA? Table 9: eight Bridge tasks x 10 trials; half successes are partial credit.
tasks = ["Eggplant in Pot (easy)", "Eggplant in Pot", "Cup into Sink", "Lift Eggplant", "Carrot on Plate", "Lift AAA Battery", "Purple Grapes", "{Eggplant, Bottle} in Pot"]
t9 = {"OpenVLA (full OpenX mixture, fused encoder)": (10, 10, 7, 7.5, 8, 7, 4, 7.5), "OpenVLA-Bridge (Bridge data only)": (8, 2, 4, 5.5, 4, 2, 3, 8), "OpenVLA-Bridge-SigLIP (Bridge only, no DinoV2)": (8, 3, 2, 6.5, 1, 2, 3, 7)}
printed = [76.3, 45.6, 40.6]
def fisher(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)
cnt = {}
for (k, v), pr_ in zip(t9.items(), printed):
    s = sum(v); cnt[k] = round(s)
    print(f"{k:48s} {s:5.1f}/80 = {100 * s / 80:5.1f}% (printed {pr_}%)")
k0, k1, k2 = t9
print(f"OpenX data effect: {100 * (sum(t9[k0]) - sum(t9[k1])) / 80:.1f} points, exact p = {fisher(cnt[k0], 80, cnt[k1], 80):.5f}")
print(f"DinoV2 effect (Bridge only): {100 * (sum(t9[k1]) - sum(t9[k2])) / 80:.1f} points, exact p = {fisher(cnt[k1], 80, cnt[k2], 80):.3f}")
print("per task, OpenVLA minus OpenVLA-Bridge:", [round(a - b, 1) for a, b in zip(t9[k0], t9[k1])])
print("per task, OpenVLA-Bridge minus OpenVLA-Bridge-SigLIP:", [round(a - b, 1) for a, b in zip(t9[k1], t9[k2])])
print("tasks where the DinoV2 model is better / equal / worse:", sum(a > b for a, b in zip(t9[k1], t9[k2])), sum(a == b for a, b in zip(t9[k1], t9[k2])), sum(a < b for a, b in zip(t9[k1], t9[k2])))
print()
print("Table 10: frozen against finetuned vision encoder (counts out of the stated trials)")
rows = [("SigLIP ViT-SO", "Put Eggplant into Pot", 10, 7, 10), ("SigLIP ViT-SO", "Put Corn on Plate", 10, 10, 9), ("LLaVA 1.5 (reproduction)", "Put Eggplant into Pot", 10, 5, 9), ("LLaVA 1.5 (reproduction)", "Put Corn on Plate", 10, 0, 9)]
rows5 = [("Eggplant/Red Bottle", 4, 2, 4), ("Blue Cup/Pink Cup", 4, 0, 0), ("Cheese/Chili", 4, 0, 3), ("Strawberry/Lime", 4, 1, 0), ("Sushi/Grapes", 4, 3, 4)]
fz = sum(r[3] for r in rows) + sum(r[2] for r in rows5); ft = sum(r[4] for r in rows) + sum(r[3] for r in rows5)
n = sum(r[2] for r in rows) + sum(r[1] for r in rows5)
print(f"  pooled over {n} rollouts: frozen {fz} ({100 * fz / n:.1f}%), finetuned {ft} ({100 * ft / n:.1f}%); printed 46.7% and 80.0%; exact p = {fisher(fz, n, ft, n):.5f}")
print(f"  mean of the three printed block averages: frozen {(85 + 30 + 25) / 3:.1f}, finetuned {(95 + 55 + 90) / 3:.1f}")
print("  five-task block with 4 trials each: frozen", sum(r[2] for r in rows5), "/20, finetuned", sum(r[3] for r in rows5), "/20 (the paper prints 30 and 55)")
print("  LLaVA frozen on Put Corn on Plate: 0/10 against 9/10 finetuned; the paper says some frozen runs were stopped for near-zero performance")

Output:

OpenVLA (full OpenX mixture, fused encoder)       61.0/80 =  76.2% (printed 76.3%)
OpenVLA-Bridge (Bridge data only)                 36.5/80 =  45.6% (printed 45.6%)
OpenVLA-Bridge-SigLIP (Bridge only, no DinoV2)    32.5/80 =  40.6% (printed 40.6%)
OpenX data effect: 30.6 points, exact p = 0.00009
DinoV2 effect (Bridge only): 5.0 points, exact p = 0.632
per task, OpenVLA minus OpenVLA-Bridge: [2, 8, 3, 2.0, 4, 5, 1, -0.5]
per task, OpenVLA-Bridge minus OpenVLA-Bridge-SigLIP: [0, -1, 2, -1.0, 3, 0, 0, 1]
tasks where the DinoV2 model is better / equal / worse: 3 3 2

Table 10: frozen against finetuned vision encoder (counts out of the stated trials)
  pooled over 60 rollouts: frozen 28 (46.7%), finetuned 48 (80.0%); printed 46.7% and 80.0%; exact p = 0.00027
  mean of the three printed block averages: frozen 46.7, finetuned 80.0
  five-task block with 4 trials each: frozen 6 /20, finetuned 11 /20 (the paper prints 30 and 55)
  LLaVA frozen on Put Corn on Plate: 0/10 against 9/10 finetuned; the paper says some frozen runs were stopped for near-zero performance

Arithmetic on the printed tables; one run per row.

Evidence in the paper

Appendix D.1 to D.3 · Tables 9 and 10

Check your understanding: Which of the three ablations is the best supported?

Finetuning the image encoder (80.0 against 46.7 over 60 rollouts, across two backbones) and using the Open X-Embodiment data (30 points over 80 rollouts). The second encoder is a suggestion, not a result.

Simulation: LIBERO (Table 12)

How it works

OpenVLA is finetuned with LoRA on each of the four LIBERO suites (Spatial, Object, Goal, Long) and compared with Diffusion Policy and Octo. Each statistic uses 500 trials over three seeds. OpenVLA averages 76.5%, Octo 75.1% and Diffusion Policy 72.4%, and has the best average rank (1.5, against 2.0 and 2.5). The authors note the margin is smaller than on real robots, which they link to OpenVLA having been pretrained on real data only.

Technical details

Our calculation Means and ranks reproduce. Per suite the order changes: Diffusion Policy is best on Object (92.5 against 88.4), Octo is best on Goal (84.6 against 79.2), and OpenVLA wins Spatial and Long. Using the printed standard errors, OpenVLA’s average lead over Octo is 1.4 points, 1.65 standard errors. Between 9% and 24% of the demonstrations were removed for failing to replay in the simulator.

Run the example: LIBERO ranks and margins

Our calculation

What each part does

Recompute the averages and ranks.
Divide each lead by its combined standard error.
Add up the demonstrations that were filtered out.

import math
# Table 12: LIBERO simulation. Three methods, four suites; success rate in percent and the printed rank within each suite.
suites = ["Spatial", "Object", "Goal", "Long"]
res = {"Diffusion Policy": ((78.3, 92.5, 68.3, 50.5), (3, 1, 3, 3), 72.4, 2.5), "Octo": ((78.9, 85.7, 84.6, 51.1), (2, 3, 1, 2), 75.1, 2.0), "OpenVLA": ((84.7, 88.4, 79.2, 53.7), (1, 2, 2, 1), 76.5, 1.5)}
se = {"Diffusion Policy": (1.1, 0.7, 1.2, 1.3, 0.7), "Octo": (1.0, 0.9, 0.9, 1.3, 0.6), "OpenVLA": (0.9, 0.8, 1.0, 1.3, 0.6)}
for k, (r, rk, avg, avgrk) in res.items():
    ranks = [1 + sum(res[o][0][i] > r[i] for o in res) for i in range(4)]
    print(f"{k:17s} mean {sum(r) / 4:6.2f} (printed {avg})   ranks recomputed {ranks} (printed {list(rk)})  mean rank {sum(ranks) / 4} (printed {avgrk})")
print()
print("OpenVLA minus the other methods, with the printed standard errors (difference / combined StdErr):")
for o in ("Diffusion Policy", "Octo"):
    for i, s in enumerate(suites + ["Average"]):
        a = (res["OpenVLA"][0][i] if i < 4 else res["OpenVLA"][2]); b = (res[o][0][i] if i < 4 else res[o][2])
        z = (a - b) / math.hypot(se["OpenVLA"][i], se[o][i])
        print(f"  vs {o:17s} {s:8s} {a - b:+5.1f} points  z = {z:+5.2f}")
print()
# demonstrations kept after replaying them in the simulator
removed = {"Spatial": 68, "Object": 46, "Goal": 72, "Long": 121}
for s, r in removed.items():
    print(f"  LIBERO-{s:8s} {500 - r}/500 demonstrations kept ({100 * r / 500:.1f}% removed because they failed to replay)")
print(f"  evaluation: 500 trials x 3 seeds = 1,500 per statistic; binomial StdErr at 76.5% over 1,500 trials = {100 * math.sqrt(.765 * .235 / 1500):.2f}")

Output:

Diffusion Policy  mean  72.40 (printed 72.4)   ranks recomputed [3, 1, 3, 3] (printed [3, 1, 3, 3])  mean rank 2.5 (printed 2.5)
Octo              mean  75.08 (printed 75.1)   ranks recomputed [2, 3, 1, 2] (printed [2, 3, 1, 2])  mean rank 2.0 (printed 2.0)
OpenVLA           mean  76.50 (printed 76.5)   ranks recomputed [1, 2, 2, 1] (printed [1, 2, 2, 1])  mean rank 1.5 (printed 1.5)

OpenVLA minus the other methods, with the printed standard errors (difference / combined StdErr):
  vs Diffusion Policy  Spatial   +6.4 points  z = +4.50
  vs Diffusion Policy  Object    -4.1 points  z = -3.86
  vs Diffusion Policy  Goal     +10.9 points  z = +6.98
  vs Diffusion Policy  Long      +3.2 points  z = +1.74
  vs Diffusion Policy  Average   +4.1 points  z = +4.45
  vs Octo              Spatial   +5.8 points  z = +4.31
  vs Octo              Object    +2.7 points  z = +2.24
  vs Octo              Goal      -5.4 points  z = -4.01
  vs Octo              Long      +2.6 points  z = +1.41
  vs Octo              Average   +1.4 points  z = +1.65

  LIBERO-Spatial  432/500 demonstrations kept (13.6% removed because they failed to replay)
  LIBERO-Object   454/500 demonstrations kept (9.2% removed because they failed to replay)
  LIBERO-Goal     428/500 demonstrations kept (14.4% removed because they failed to replay)
  LIBERO-Long     379/500 demonstrations kept (24.2% removed because they failed to replay)
  evaluation: 500 trials x 3 seeds = 1,500 per statistic; binomial StdErr at 76.5% over 1,500 trials = 1.09

Arithmetic on the printed table and standard errors; it ignores correlation between suites.

Evidence in the paper

Appendix E · Table 12

Check your understanding: Why does OpenVLA have the best rank but not the best score in every suite?

Rank is the average position across suites, so winning two suites and placing second in the other two beats winning one. The per-suite order changes from suite to suite, which suggests no method is uniformly best.

What the results show

A 7-billion-parameter VLA trained end to end on 970 thousand robot episodes is far better than RT-1-X, Octo and a model finetuned without robot pretraining on the Bridge robot, and about level with RT-2-X on the Google robot.
Finetuning the image encoder and pretraining on a broad mixture matter: the two large ablation effects (about 30 points).
OpenVLA is a good default for finetuning on tasks with several objects and instructions, and LoRA with 4-bit serving brings it to one consumer-grade card without a visible loss.

What they do not show

That the 16.5% lead over RT-2-X is general: it is 20 points on one robot and within chance on the other, with different data cleaning and a workaround for RT-2-X.
That the fused encoder, LoRA against full finetuning or 4-bit against bfloat16 differ by anything beyond chance; those tests use 33 to 80 rollouts of a variant model.
That OpenVLA beats Diffusion Policy on narrow tasks, or Octo on tasks with several objects: it does not.
Reliability: the authors say success typically stays below 90%, and the model sees one image, no history and no chunk.

Part E · optional

Engineering deep dive

The tokeniser, the hyperparameters, the data mixture, the evaluation tasks and the released code, for readers who want to rebuild or audit the work. Everything is collapsed; Parts A to D do not depend on it.

The action tokeniser (Section 3.2)

Reported by the paper For each of the seven action dimensions, take the 1st and 99th percentile q_lo and q_hi of that dimension in the training data and cut the interval into 256 equal bins. A value a becomes the bin index below, after clipping to the interval:

b=\frac{\lfloor 256\cdot (\operatorname{clip} (a,\,q_{\text{lo}},\,q_{\text{hi}})-q_{\text{lo}})}{(q_{\text{hi}}-q_{\text{lo}})\,\rfloor },\,\text{token id}=(\text{vocabulary size}-256)+b

The formula is our rendering of the description (the paper gives no equation). The Llama tokenizer reserves only 100 special tokens, so the 256 least-used tokens, which are the last 256, are overwritten. The model predicts the seven tokens autoregressively with the usual cross-entropy loss on those tokens only. At run time, each token is decoded to the centre of its bin and the number is rescaled with the dataset statistics of the target robot.

Hyperparameters and infrastructure

As printed in Sections 3.4, 3.5, 5.2 and 5.3.
Setting	Value
Backbone	Prismatic-7B: SigLIP and DINOv2 (about 600 million parameters together), 2-layer MLP projector, Llama 2 7B
Input	one third-person image, 224 × 224 pixels; no history, no proprioception
Action	7 numbers, 256 bins each between the 1st and 99th percentile; the 256 least-used tokens are overwritten
Loss	cross-entropy on the action tokens only
Learning rate	2e-5, fixed, no warm-up
Batch size, epochs	2,048; 27 passes over the data (until action-token accuracy exceeds 95%)
Pretraining compute	64 A100 cards for 14 days, 21,500 card-hours
Inference	bfloat16: 15 GB, about 6 Hz on one RTX 4090; 4-bit: 7.0 GB (Table 2); remote inference server released
Full finetuning	8 A100 cards for 5 to 15 hours per task, 10 to 150 demonstrations
LoRA finetuning	rank 32 on all linear layers: 10 to 15 hours on one A100; rank 64 gives the same result

The training mixture (Table 3)

As printed, sorted by share. Our count of the rows is 27 and the shares add up to about 94.9%.
Dataset	Share
Bridge	13.3%
Fractal	12.7%
Kuka	12.7%
DROID	10.0% (dropped for the last third of training)
BC-Z	7.5%
FMB	7.1%
Language Table	4.4%
Stanford Hydra	4.4%
Taco Play	3.0%
Furniture Bench	2.4%
Roboturk	2.3%
Austin Sailor	2.2%
UTAustin Mutex	2.2%
Toto	2.0%
Austin Sirius	1.7%
DobbE	1.4%
Berkeley Autolab UR5	1.2%
Viola	0.9%
IAMLab CMU Pickup Insert	0.9%
NYU Franka Play	0.8%
Berkeley Fanuc	0.7%
Jaco Play	0.4%
Berkeley Cable Routing	0.2%
Austin Buds	0.2%
CMU Stretch	0.2%
UCSD Kitchen	< 0.1%
DLR EDAN	< 0.1%

The evaluation tasks (Appendix B)

Reported by the paper BridgeData V2 WidowX (17 tasks, 10 trials): five visual tasks (eggplant into pot in an easy and a normal version, a cup from the counter into the sink, eggplant with clutter, yellow corn on a plate), two motion tasks (lift eggplant, carrot on a raised plate), three physical (carrot on plate, flip pot, lift a battery), four semantic (skull into a rack, lift white tape, grapes out of a pot, stack cups) and three language-grounding tasks, run as pairs with the same starting scene and two different prompts. In 16 of 17 tasks the arm starts at a fixed place above the sink, not above the target, which the original data never did in this scene.
Reported by the paper Google robot (12 tasks, 5 trials): five in-distribution tasks (pick a can, move an apple near a can, move a chip bag near an apple, place a can upright, open a drawer) and seven out-of-distribution (new objects, backgrounds, instructions such as placing the apple on a plate, and moving a can to a photo of a celebrity).
Reported by the paper Franka-Tabletop (5 Hz) and Franka-DROID (15 Hz): carrot into a bowl (50 demonstrations), pour corn into a pot (50), flip a pot (10), move an object onto a plate (150), knock an object over (70), cover an object with a towel (45) and wipe a table with a brush (70), each with in-distribution and out-of-distribution variants. The Diffusion Policy baseline also uses a two-step image and proprioception history, predicts absolute coordinates and acts on a chunk of 8 or 16 actions; the matched version uses a single image and a single relative action.
Reported by the paper Scoring: success 1 or failure 0, with partial credit of 0.5 for several tasks (moving towards the right object, touching it), and 0, 1 or 2 points for the table-wiping task.

The released code

Reported by the paper Section 4: a PyTorch codebase that scales from one card to multi-node clusters with automatic mixed precision, FlashAttention and fully sharded data parallelism, supports training on Open X-Embodiment, integrates with the Hugging Face model classes, and supports LoRA finetuning and quantised inference. A remote inference server streams actions to the robot. Checkpoints and finetuning notebooks are released.

Limitations stated by the authors (Section 6)

Only single-image observations: no extra cameras, proprioception or history.
Inference throughput is too low for high-frequency robots such as ALOHA at 50 Hz; action chunking or speculative decoding are suggested.
Reliability is not high: success is typically below 90%.
Open questions, left for lack of compute: the effect of the language model’s size, co-training with web vision-language data, and which visual features work best.

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: where its numbers disagree, where a comparison differs in more than the thing being credited, and where small sample sizes limit the conclusions. OpenVLA is an open, reproducible and useful piece of work, and its tables are unusually complete, which is what makes the re-checks possible. The findings concern the strength of the evidence. Each one links back to the concept it affects, and the calculations are reproducible in the examples.

Numbers that do not agree

F1. The training mixture adds up to 94.9% Numerical inconsistency

What the paper says. Table 3 lists the share of each dataset in the training mixture, “following” Octo with a few additions.

What we found. The 27 listed shares add up to 94.9% (with “<0.1%” counted as 0.05), so about 5 points are unaccounted for; Octo’s own table, with 25 datasets, adds up to about 99.8%. The three largest datasets are 38.7%, against 51.0% in Octo’s mixture. The footnote says DROID was dropped for the last third of training and its weight redistributed, so the table is not the mixture of the whole run either.

Why it matters. A reader who wants to reproduce the mixture cannot from the table alone, and the share of each dataset over the full run is not what is listed. Concept: The data · example
F2. “16.5% over 29 tasks” is a pooled number that hides two very different gaps Numerical inconsistency

What the paper says. The abstract says OpenVLA outperforms RT-2-X by 16.5% in absolute task success across 29 tasks and several embodiments.

What we found. Pooling the rollouts (170 on Bridge, 60 on the Google robot) reproduces it: 74.35% against 57.83%. But the pooled gap is mostly the Bridge robot, where the difference is 20.0 points (120 against 86 successes of 170); on the Google robot it is 6.7 points (51 against 47 of 60, exact p of 0.480). The unweighted mean of the two robot gaps is 13.3 points. RT-2-X is slightly ahead on the semantic tasks, which the text acknowledges but the abstract omits.

Why it matters. The headline is a rollout-weighted average that depends on how many tasks each robot got; it is not evidence that OpenVLA beats RT-2-X in general. Concept: Direct evaluations · example
F3. The “20.4%” gain over Diffusion Policy is not reproduced, and it is specific to multi-instruction tasks Numerical inconsistency

What the paper says. The abstract says finetuned OpenVLA outperforms Diffusion Policy by 20.4%, with especially strong results in multi-task settings with multiple objects and language grounding.

What we found. From Table 7 the gain is 19.8 points pooled over the 129 rollouts, 21.0 as the mean of the two setup averages and 17.1 as the mean of the seven task averages; no aggregation gives 20.4. By task group, OpenVLA is 44 points ahead on the three multi-instruction tasks (65.7 against 21.3), but behind Diffusion Policy on the three narrow ones (68.9 against 81.1; the gap is within chance with 45 rollouts, p of 0.334).

Why it matters. The sentence is a mix of an average and a qualifier; the useful message is the split, which the main text states but the abstract compresses to one number. Concept: Finetuning · example
F4. The Bridge standard errors are smaller than a binomial calculation gives Numerical inconsistency

What the paper says. Figures and tables report mean success ± StdErr across 170 Bridge rollouts per method.

What we found. For Bridge, the printed standard errors (2.7, 2.6, 3.5, 3.2) are 10 to 15% below the binomial values for the same means and 170 rollouts (3.0, 3.1, 3.8, 3.5). On the Google robot the printed values agree with a binomial calculation on 60 rollouts (6.1, 5.8, 5.4, 4.6, within rounding of the n−1 form). The formula is not stated. Partial credit (0.5) makes the scores less variable than success or failure, which may explain part of it.

Why it matters. The difference is small and does not change a conclusion, but it means the error bars are not the plain binomial ones a reader would assume. Concept: Direct evaluations · example
F5. “An 8x reduction in compute” holds at one end of the range Numerical inconsistency

What the paper says. LoRA finetuning on one A100 takes 10 to 15 hours, an 8x reduction in compute compared with full finetuning.

What we found. Full finetuning used 8 A100 cards for 5 to 15 hours depending on dataset size, which is 40 to 120 card-hours. Against a 10-hour LoRA run that is 4 to 12 times, and against 15 hours 2.7 to 8 times. The paper does not say which full-finetuning duration belongs to which LoRA duration.

Why it matters. The saving is real and large, but “8x” is the paper’s rounding of a range from about 3 to 12. Concept: LoRA · example

How far the claims go

F6. The comparison with RT-2-X mixes data, cleaning, encoder and a workaround Method caveat

What the paper says. OpenVLA outperforms RT-2-X by an order of magnitude fewer parameters, which the authors attribute to a larger and cleaner dataset and a fused encoder.

What we found. RT-2-X is a closed model reached through an interface the authors had early access to, and could not be retrained. Its training data were smaller (350 thousand episodes) and not cleaned of the all-zero first action in Bridge, so for the Bridge tests it is asked for its second most likely action, a workaround the authors say is still worse than retraining. The paper’s only ablation of the three suggested causes is Table 9, which shows the data effect (30 points) is large and the encoder effect (5 points) is not distinguishable from chance. Data cleaning is not ablated at all. On the Google robot, where this workaround is not needed, the gap is within chance.

Why it matters. The result supports “a well-trained open VLA can match or beat RT-2-X”; it does not support a ranking of the design choices, and the 50.6% for RT-2-X on Bridge is partly a statement about how it was queried. Concept: Cleaning the data · Direct evaluations · What matters
F7. The LoRA and quantisation results use a different, smaller model Method caveat

What the paper says. LoRA matches full finetuning with 1.4% of the parameters, and 4-bit serving matches bfloat16 without a hit to success.

What we found. Footnote 4 says Sections 5.3 and 5.4 use a version pretrained on the smaller, Octo-sized mixture with a SigLIP-only encoder. Its bfloat16 score on the eight Bridge tasks (57 of 80) is 4 successes below the released model on the same tasks (61 of 80, Table 9). The LoRA comparison is 33 rollouts of two Franka tasks; full finetuning (23 successes) cannot be separated from LoRA (22), from the sandwich variant (21), and only barely from a frozen vision encoder (15, p of 0.080).

Why it matters. The claims are plausible and likely to hold, but the released model was not the one tested, and the sample cannot show “no loss”, only “no large loss”. Concept: LoRA · Quantisation · example · example
F8. Run-to-run noise is visible in the paper’s own tables Method caveat

What the paper says. Success rates are compared across methods, precisions and ablations, with 5 to 10 trials per task and one run each.

What we found. The same bfloat16 model on the same eight tasks scores 57 and 56 of 80 under the two control modes, but single tasks move by 4 to 6 trials (Purple Grapes 8 to 2, Lift AAA Battery 7 to 3, Carrot on Plate 6 to 10). With 10 trials a rate of 5/10 has a 95% interval of 24% to 76%. With 80 rollouts the smallest difference an exact test calls real is about 18 points, and with 33 rollouts 27.

Why it matters. The large effects (data, pretraining, an unfrozen image encoder, last-layer-only finetuning) are safe; the many smaller ones (fused encoder, 8-bit, rank 32 against 64) are suggestions. Concept: The set-up · Quantisation · example · example
F9. The Diffusion Policy baseline differs in more than the pretraining Method caveat

What the paper says. OpenVLA is a better default than a from-scratch imitation learner; it is the only approach with at least 50% success on every task.

What we found. The full Diffusion Policy uses a two-step history, proprioception, absolute coordinates and action chunking, which OpenVLA lacks; the matched version removes these and scores lower (43.4% against 48.5%). The “at least 50% on every task” claim holds for the pooled task rates (OpenVLA’s lowest is exactly 50.0% on Cover with Towel, over 18 rollouts), but a finetuned Octo is far better on that task (91.7% against 50.0%). OpenVLA and Octo are level on the multi-instruction tasks (35 and 34 of 54), so the advantage over Octo comes from the narrow tasks and Franka-DROID.

Why it matters. The result shows OpenVLA is a reasonable default, not that a large VLA is better than a small policy for tasks with several objects, where a finetuned 93-million-parameter Octo does as well. Concept: Finetuning · example
F10. The fused-encoder explanation rests on a text comparison and a result within chance Limitation

What the paper says. The DINOv2 half of the encoder gives better spatial reasoning, which is useful for robot control.

What we found. The evidence is a roughly 10-point gain of Prismatic over a LLaVA-based policy, stated in the text without a table, where the two backbones differ in more than the encoder, and Table 9, in which removing DINOv2 from a Bridge-only model costs 5 points (36 against 32 successes of 80, p of 0.632). DINOv2 helps on 3 tasks, ties on 3 and hurts on 2. No experiment tests spatial reasoning directly.

Why it matters. The fused encoder is a reasonable design, but the claim that it is responsible for better spatial control is untested. Concept: Two image encoders · What matters · example
F11. On LIBERO the lead is small and the order changes by suite Method caveat

What the paper says. Finetuned OpenVLA obtains the highest average success rate and rank among the tested methods on LIBERO.

What we found. It is true for the mean (76.5 against 75.1 for Octo and 72.4 for Diffusion Policy) and the mean rank. But Diffusion Policy is best on Object and Octo on Goal, the lead over Octo is 1.4 points (1.65 standard errors, ignoring correlation), and the comparison with Octo differs in LoRA against full finetuning. The demonstrations were also filtered (9% to 24% removed), for all methods alike.

Why it matters. The result supports “comparable to the best baselines in simulation”; the authors, to their credit, say the margin is tighter than in the real-world experiments. Concept: Simulation · example

What is not tested

F12. “First” claims that the paper cannot check Ambiguity

What the paper says. OpenVLA is the first generalist VLA that is open-source, and the first to demonstrate parameter-efficient finetuning and quantisation for VLAs.

What we found. The paper offers no survey of earlier models, and its own footnote notes that the weights of SigLIP, DINOv2 and Llama 2 are open but their training data and code are not; what the authors release is the data mixture, the code and the weights of the combination. Octo, which they compare against, is also open and finetunable, so the distinction is that OpenVLA is the first open model that starts from a vision-language model.

Why it matters. The release is concrete and checkable; the “first” statements are positioning at the time of writing. Concept: The vision-language model · LoRA
F13. What the paper says about its limits, and what it leaves out Limitation

What the paper says. The authors list single-image input, low throughput, success below 90%, and open design questions such as model size and co-training.

What we found. These are accurate. The paper does not discuss that every number is a single run, that the dataset cleaning is explained only in appendix prose, that RT-2-X was co-trained with web data and OpenVLA was not (which may explain the semantic tasks), that the throughput limit follows from writing seven tokens one by one, or that its finetuning experiments are all on one arm type (Franka) with a single camera. It also trains with a single-step policy and so cannot say whether action chunking would change the Diffusion Policy comparison.

Why it matters. OpenVLA is a solid open starting point for finetuning; the evidence supports “finetunes well on one camera and a single step” more than “generalises to new tasks and embodiments out of the box”. Concept: What is trained · Direct evaluations · Finetuning

Knowledge check

Try each question first, then reveal the explanation.

Question 1

In your own words: how does OpenVLA produce a robot action from an image and a sentence?

Explanation

A pretrained vision-language model reads the image through two joined encoders and the sentence as text, then writes seven tokens, one at a time. Each token is the bin index of one action number; the bins are mapped back to the centre of each range and rescaled to robot units.

Question 2

Why are the action tokens written over the 256 least-used words?

Explanation

The Llama tokenizer reserves only 100 spare tokens, which is too few for 256 bins, so the last 256 vocabulary entries are overwritten. It is the simplest way to add an action vocabulary without changing the model.

Question 3

What do the finetuning and ablation tables show, and what do they not?

Explanation

Open X-Embodiment pretraining and an unfrozen image encoder matter by about 30 points; OpenVLA is a good default for tasks with several objects. They do not show that the fused encoder matters, that LoRA is exactly as good as full finetuning, or that OpenVLA beats Diffusion Policy on narrow tasks.

Question 4

Name one place where the paper’s numbers or claims need a closer look.

Explanation

For example: the Table 3 shares add up to 94.9%; the abstract’s 16.5% gain over RT-2-X is 20 points on one robot and within chance on the other; or the abstract’s 20.4% gain over Diffusion Policy is not reproduced by Table 7.

Glossary

Vision-language-action (VLA) model: A vision-language model finetuned to output robot actions.
Vision-language model (VLM): A model that reads images and text and writes text.
Prismatic: The open vision-language model family that OpenVLA starts from: two image encoders, a projector and a Llama 2 language model.
SigLIP: An image encoder trained to match images with captions; strong on meaning.
DINOv2: An image encoder trained without captions; known for spatial detail.
Projector: A small network that maps image features into the language model’s input space.
Action token: A word of the language model’s vocabulary that stands for one bin of one action number.
Quantile bins: Equal-width bins between the 1st and 99th percentile of the training actions.
Next-token prediction: The usual language-model training: predict each token given the ones before it.
Open X-Embodiment: A pooled collection of robot datasets from many labs.
Bridge and Google robot: The two robots used in the direct tests: a WidowX arm in a sink scene and the Google mobile manipulator.
RT-2-X: A 55-billion-parameter VLA of Google DeepMind finetuned on Open X-Embodiment data; closed.
Diffusion Policy: An imitation learner that generates action chunks by denoising; trained from scratch for each task.
Low-rank adaptation (LoRA): Finetuning by training small extra matrices instead of the original weights.
Quantisation: Storing the weights in fewer bits (8 or 4) to save memory.
bfloat16: A 16-bit number format; OpenVLA’s default for serving.
Blocking control: The robot waits for each prediction before moving on; removes the effect of model speed.
No-op action: An all-zero action; imitating it can make a policy freeze.
LIBERO: A simulated manipulation benchmark with four task suites.
Standard error: How much a measured rate would typically move by chance if the trials were repeated.
Trial: One attempt at a task; success rates here are out of 5 to 170 trials.

References

Numbered notes in the text, like ^[1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

Kim, M. J., et al. (2024). OpenVLA: An open-source vision-language-action model. Conference on Robot Learning (CoRL) 2024. arXiv:2406.09246
Cited for:
- A 7-billion-parameter open-source VLA trained on 970,000 real-world robot demonstrations, built on a Llama 2 language model with a visual encoder that fuses pretrained DINOv2 and SigLIP features. Checked against the paper’s abstract.
- Each action dimension is split into 256 bins between the 1st and 99th quantile of the training actions; the 256 bins are written over the 256 least-used tokens of the Llama tokenizer, and the model is trained with next-token prediction on the action tokens. Checked against the paper, via summaries of its text.
- OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Checked against the paper’s abstract.

Go further

All paper guides · Back to the VLA track

How to use this page

Start here: the paper at a glance

The problem

The main contribution

How the model works, in one paragraph

What it learns from, and what it is tested on

What the paper reports

What to keep in mind

Understand the method

1. The starting point: a vision-language model

2. Two image encoders instead of one

3. Writing an action as seven tokens

4. What is trained, and how the policy runs

Understand the data and the training

The data: 970 thousand episodes from Open X-Embodiment

Cleaning the data: why a policy can freeze

The training recipe: what small experiments decided

Understand the experiments

The set-up: robots, tasks, comparisons and trial counts

Direct evaluations: OpenVLA against RT-2-X, Octo and RT-1-X (Tables 4 and 6)

Finetuning to a new robot (Table 7)

Cheaper finetuning: LoRA (Tables 1 and 8)

Cheaper serving: 8-bit and 4-bit (Tables 2, 5 and 11)

What matters: Open X-Embodiment data, the second encoder, finetuning the encoder (Tables 9 and 10)

Simulation: LIBERO (Table 12)

What the results show

What they do not show

Engineering deep dive

Critical paper review

Numbers that do not agree

F1. The training mixture adds up to 94.9% Numerical inconsistency

F2. “16.5% over 29 tasks” is a pooled number that hides two very different gaps Numerical inconsistency

F3. The “20.4%” gain over Diffusion Policy is not reproduced, and it is specific to multi-instruction tasks Numerical inconsistency

F4. The Bridge standard errors are smaller than a binomial calculation gives Numerical inconsistency

F5. “An 8x reduction in compute” holds at one end of the range Numerical inconsistency

How far the claims go

F6. The comparison with RT-2-X mixes data, cleaning, encoder and a workaround Method caveat

F7. The LoRA and quantisation results use a different, smaller model Method caveat

F8. Run-to-run noise is visible in the paper’s own tables Method caveat

F9. The Diffusion Policy baseline differs in more than the pretraining Method caveat

F10. The fused-encoder explanation rests on a text comparison and a result within chance Limitation

F11. On LIBERO the lead is small and the order changes by suite Method caveat

What is not tested

F12. “First” claims that the paper cannot check Ambiguity

F13. What the paper says about its limits, and what it leaves out Limitation

Knowledge check

Question 1

Question 2

Question 3

Question 4

Glossary

References

Go further