6.1 — OpenVLA

Learning objectives

After this lesson, you should be able to:

• Describe OpenVLA’s three components and how it outputs actions.
• State its reported comparisons with their units.
• Estimate the memory for model weights at different precisions.
• Explain why low-rank adaptation needs few trained parameters.

The one-sentence idea

OpenVLA is a language model taught to write action tokens after looking at a picture.

Open, 7B, token actions; fine-tunable with low-rank methods.

An open VLA you can download

RT-2 showed the idea, but its model was closed. OpenVLA is an open-source VLA with 7 billion parameters, trained on 970,000 real-world robot demonstrations. It builds on a Llama 2 language model combined with a visual encoder that fuses pretrained DINOv2 and SigLIP features.^[1] It is therefore the best model to read about first: its structure is public and matches the pipeline of lesson 1.4.

Its three parts are a visual encoder that turns the image into patch embeddings, a projector that maps them into the language model’s input space, and the language-model backbone.^[1] Actions are tokens: each action dimension is split into 256 bins between the 1st and 99th quantile of the training actions, the bins are written over the 256 least-used tokens of the Llama tokenizer, and the model is trained with next-token prediction on the action tokens.^[1] At run time the tokens are decoded to continuous values and de-normalised to the robot’s physical scale.^[1]

What the paper reports

Two headline comparisons, both from the abstract. Against the closed 55-billion-parameter RT-2-X, OpenVLA is better by 16.5 percentage points of absolute task success across 29 tasks and several robot embodiments, with 7 times fewer parameters.^[1] And in fine-tuning experiments it outperforms the from-scratch imitation method Diffusion Policy by 20.4% in multi-task settings with multiple objects and language grounding.^[1]

Note how these claims are phrased: a unit (absolute success), a set (29 tasks), a baseline (RT-2-X). Compare the advice in lesson 5.5. For the full tables, a re-check of the arithmetic (the 16.5 points are 20 on one robot and within chance on the other) and a critical review, see the OpenVLA paper guide.^[1]

Can I run a 7-billion-parameter model?

The abstract also states that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate.^[1] Two bits of arithmetic explain both. First, the memory just to hold the weights:

params = 7_000_000_000
for name, bytes_per_weight in (("32-bit floats", 4), ("16-bit floats", 2), ("8-bit integers", 1), ("4-bit integers", 0.5)):
    print(f"{name:<15} -> {params * bytes_per_weight / 1e9:5.1f} GB")

32-bit floats   ->  28.0 GB
16-bit floats   ->  14.0 GB
8-bit integers  ->   7.0 GB
4-bit integers  ->   3.5 GB

Toy model. Weights only. Running the model also needs memory for activations and the cache, so real requirements are higher.

Dropping from 16 to 8 or 4 bits halves or quarters the weight memory. That is quantisation. Does it damage the numbers? Here is rounding to 8-bit integers on a random weight matrix:

import numpy as np
rng = np.random.default_rng(0)

W = rng.normal(0, 0.02, (512, 512))                         # a weight matrix
x = rng.normal(size=512)

scale = np.abs(W).max() / 127                                # map the largest weight to 127
W_int8 = np.round(W / scale).astype(np.int8)
W_back = W_int8.astype(np.float64) * scale

y, y_q = W @ x, W_back @ x
print("largest weight error:", round(float(np.abs(W - W_back).max()), 5))
print("relative error of the output:", round(float(np.linalg.norm(y - y_q) / np.linalg.norm(y)), 5))

largest weight error: 0.00037
relative error of the output: 0.01088

The output changes by about 1%. Whether the robot still succeeds is a separate question; the paper reports no hit to downstream success for its setting.

Fine-tuning without touching every weight: low-rank adaptation

Fine-tuning all 7 billion weights needs far more memory than holding them. Low-rank adaptation (LoRA) freezes the weights and learns a small correction for selected matrices: instead of changing a d × d matrix, learn two thin matrices of shapes d × r and r × d whose product is the correction. The count of trained numbers drops dramatically:

d = 4096                                   # the width of one weight matrix, as an example
for r in (4, 8, 32):
    full = d * d
    lora = 2 * d * r
    print(f"rank {r:>2}: {lora:>9,} trained numbers instead of {full:,} ({100 * lora / full:.2f}%)")

rank  4:    32,768 trained numbers instead of 16,777,216 (0.20%)
rank  8:    65,536 trained numbers instead of 16,777,216 (0.39%)
rank 32:   262,144 trained numbers instead of 16,777,216 (1.56%)

Toy model. The width 4096 is an example size, and which matrices get a correction is a design choice.

That is why a consumer GPU can fine-tune it: the optimiser only has to track the small corrections. We use this in lesson 7.4.

What it does not do

OpenVLA predicts one action per step as tokens, which means the limits from lesson 4.2 apply: fine resolution is bounded by 256 bins, and long chunks are costly. The FAST paper argues that per-dimension, per-timestep binning performs poorly on dexterous, high-frequency data.^[2] That is a main reason newer models (lessons 6.3 and 6.4) use other action outputs.

Common misconceptions

Engineering takeaways

1. 970,000 demonstrations; Llama 2 plus DINOv2 and SigLIP.
2. 256 bins per action dimension.
3. Quantisation cuts memory; LoRA cuts trained parameters.
4. Report comparisons with units.

OpenVLA: the open recipe for a token-based VLA.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: Octo, a generalist policy built to be fine-tuned to new robots.

Back to the VLA roadmap