Skip to the walkthrough
FixTheRobot Home / VLA / Paper guides / Octo

VLA · Paper guide

Octo: An Open-Source Generalist Robot Policy

A small, open generalist policy that is meant to be finetuned, explained through the questions an engineer would ask.

  • Octo Model Team (Ghosh, Walke, Pertsch, Black, Mees et al.)
  • arXiv:2405.12213 (v2)
  • Overview: about 4 minutes
  • Full guide: about 50 minutes

This is a guide to the Octo paper, not a sentence-by-sentence paraphrase. Octo is a robot policy built before the flow-matching VLAs in the other guides: it has no vision-language model, but it is built around a token interface that new robots can plug into, and it uses a diffusion head, which later models such as π0 replaced with flow matching. Part A is a 4-minute overview. Parts B to D explain the method, the data and the experiments one idea at a time. Equations and tables are folded away in Part E, and Part F is a critical review of the paper’s numbers and claims.

Start with the overview Jump to the critical review

How to use this page

  1. In a hurry: read Part A (about 4 minutes).
  2. To understand the method: add Parts B to D. Each idea follows the same pattern: intuition, the problem it solves, how it works, an example, then technical details and where to find them in the paper.
  3. To check or rebuild it: open the folded sections (the denoising step, hyperparameters, the mixture, runnable examples) and read Part F.

Open the paper (arXiv:2405.12213) beside this page; section, table and figure references are to it. We paraphrase and never reproduce its text. Statements are labelled so you can tell them apart: Reported by the paper, Our calculation from the paper’s numbers, and Our interpretation. Runnable examples are marked Toy illustration when they are our own small model of an idea, and are never presented as results of the paper.

Part A · about 4 minutes

Start here: the paper at a glance

The problem

Robot policies are usually trained from scratch for one robot and one task, which needs new data every time and gives narrow skills. Models for language and images became useful because one model, pretrained on broad data, could be adapted cheaply. The authors ask for the robot version: a generalist robot policy (GRP), pretrained on many robots, that controls several of them out of the box and can be finetuned with a little data to a robot with different cameras, sensors or action format. They note that earlier multi-robot models fixed the inputs they accept, did not finetune easily to new ones, and often were not released.

The main contribution

  1. Octo, a transformer policy with a flexible interface. Instructions, goal images and camera images are all turned into tokens, so a different set of cameras or a different way of saying the task only changes which tokens go in. New inputs and new outputs can be attached later without retraining the pretrained part.[1]
  2. A very broad, open pretraining set. 800 thousand robot episodes from 25 datasets of Open X-Embodiment, the largest manipulation collection the authors know of at the time, with the data pipeline, checkpoints and code released.[1]
  3. Evidence from 9 real robot setups and design ablations. Zero-shot control, finetuning to new observations, new action formats and new robots, and a set of ablations on architecture, data, objective and size.[1]

How the model works, in one paragraph

The task, a sentence or a goal image, and the camera images of the last two time steps become a sequence of tokens: the sentence goes through a frozen pretrained text encoder (T5-base), and each image goes through a shallow convolutional network and is cut into small patches. A transformer reads this sequence with a masking pattern that lets each time step see the task and the past but nothing else. Extra “readout” tokens collect the result without influencing anything. A small diffusion head, a network that starts from random noise and removes it in 20 small steps, turns each readout into a chunk of several future actions. Only one pass of the large transformer is needed per chunk; the 20 steps run inside the small head.

What it learns from, and what it is tested on

Learns from: imitation of the demonstrations in 25 Open X-Embodiment datasets (a mix of robots, scenes and tasks; the three largest datasets make up half of every batch), in two sizes: Octo-Small with 27 million parameters and Octo-Base with 93 million. Tested on: nine real-robot setups at four institutions. Three are used as they are (zero-shot), with tasks taken from the pretraining datasets; six are finetuned with about 100 demonstrations each, including new sensors (force and torque readings), a new action format (joint positions) and new robots (including a two-armed one). Comparisons are with two earlier generalist models, RT-1-X and RT-2-X, for zero-shot use, and with a policy trained from scratch and a pretrained visual representation (VC-1) for finetuning.

What the paper reports

Section IV and Tables I and II, as printed. The zero-shot numbers are not printed in the text.
TestOctoComparison
Zero-shot, three robots (success)above RT-1-X by 29% on average; similar to RT-2-Xvalues shown only in a bar chart
Finetuning, six setups, about 100 demonstrations (average, %)7220 (ResNet and transformer, from scratch); 15 (VC-1)
Ablations on the WidowX setup (average, %)83 for the chosen design18 discretised actions; 35 mean-squared-error head; 70 ResNet-50 encoder; 60 smaller data mix
Model size and zero-shot successimproves from 10 to 27 to 93 million parametersone task per robot, 10 trials

What to keep in mind

  • The sample sizes are small: 10 trials per task for the zero-shot and scaling tests, 20 per finetuning setup and 40 per ablation row. Several gaps, such as the vision transformer (ViT) against a ResNet encoder in the ablation, are within chance.
  • The finetuning baselines use different architectures and loss functions from Octo, so “pretrained against scratch” is not a clean comparison.
  • Octo does not start from a vision-language model, and the authors say it handles wrist cameras and language less well than it should, which they link to missing data.
  • A few numbers and counts in the paper disagree with each other (Part F).

If you read nothing else: Octo shows that one small transformer, pretrained on 800 thousand episodes with a token interface and a diffusion head, can be used on several robots and finetuned in a few hours to new sensors, action formats and robots with about 100 demonstrations; the evidence is real but rests on small trial counts and baselines that differ in more than the pretraining.

Check your understanding: Why does Octo turn instructions, goals and images into tokens in the same sequence?

Because then a different robot setup only changes which tokens are present. Missing inputs are masked out and new ones are added with a small new encoder, so the pretrained transformer does not need to be rebuilt.

Part B

Understand the method

Octo’s architecture is a tokenizer for every kind of input, a transformer with a careful masking pattern, readout tokens and a small diffusion head. The design goal is flexibility: the same pretrained network should accept different inputs and outputs. Each idea below follows the same pattern, from intuition to where to find it in the paper.

How Octo turns a task and camera images into a chunk of actions A text instruction goes through a frozen text encoder and a goal image or camera images go through a shallow convolutional network that cuts them into patches. All tokens enter one transformer with block-wise masking. Readout tokens at the end of each time step collect the result. A small diffusion head turns the readout into a chunk of actions. For a new robot, new encoders or a new action head can be attached without changing the transformer. 1. Task and observationsa sentence or a goal image, and camera imagesfrom the last two time steps 2. Turn everything into tokenstext: frozen T5-base encoder, 16 tokensimages: shallow CNN, 16 x 16 patches 3. One transformer, block-wise maskingeach step sees the task and the past onlymissing inputs are masked out 4. Readout tokenscollect the result, but nothing reads themso other tokens are not disturbed 5. Diffusion action head (small)noise to a chunk of actions in 20 small stepsone transformer pass per chunk Adapting to a new robotadd a new encoder, a new position embeddingor a new action head; the pretrainedtransformer stays as it is
One prediction. Orange marks the parts the guide calls out as the paper’s main design choices. Drawn by us from Section III and Figure 2 of the paper.

1. Everything becomes tokens

Intuition
Different robots give different inputs; turning every input into the same kind of token lets one network read them all.
The problem it solves
Earlier multi-robot models fixed the observations they accepted, for example one camera, so using them on a different setup meant retraining large parts of the model.
How it works
The instruction goes through a pretrained text encoder (T5-base, 111 million parameters, frozen) and becomes 16 language tokens. Images, whether camera observations or goal images, go through a shallow convolutional network and are cut into 16 by 16 patches: 256 tokens for a 256 by 256 third-person image and 64 tokens for a 128 by 128 wrist image. Learned position embeddings are added, and the tokens are arranged in a sequence: task tokens first, then the observation tokens of each time step. Octo uses two time steps of history.
Example
Illustration, not from the paper A robot with one third-person camera and a sentence gives 256 + 16 tokens; a robot with a wrist camera as well adds 64 more. The transformer is the same for both.
Why the authors chose it
The authors say the choice lets one pretrained model serve many setups, by changing which tokens are fed in.
Technical details
Goal images are an alternative to language: during training either the instruction or the goal image is randomly removed for each example, so one model can be told the task either way, and for datasets without language the goal image is always used.
Evidence in the paper
§III-A · §III-D · Appendix D
Check your understanding: Why are the image tokens made from patches and not from a large pretrained image network?

The authors chose a transformer-first design with a very shallow convolutional stem, so that most of the computation sits in the transformer; their ablation reports it scales better on the large mixture than a ResNet encoder.

2. A transformer with block-wise attention

Intuition
A seating plan that says who may listen to whom, so inputs can be added or removed without upsetting the others.
The problem it solves
If every token can see every other token, adding a camera or removing the language instruction changes what everything else computes.
How it works
The attention is masked in blocks. Observation tokens may look at the task tokens and at tokens of the same or earlier time steps, not at later ones. Tokens of a missing input, such as the language of a dataset without instructions, are masked out completely. Task tokens look only at task tokens.
Example
Illustration, not from the paper When the second camera frame arrives, it can read the first frame and the instruction, but the first frame never reads the second.
Why the authors chose it
The structure makes the model modular, which is what allows new inputs to be attached when finetuning.
Technical details
Toy illustration The example builds this pattern with random weights, checks that changing a later observation does not change an earlier one, and shows a sequence that still runs without the task tokens.
Run the example: block-wise attention in miniature
Toy illustration

What each part does

  1. Build the mask: task, observation and readout tokens over two time steps.
  2. Change one token at a time and see which outputs move.
  3. Mask out the task tokens and run the sequence again.
import numpy as np

# Octo's block-wise attention in miniature (Section III-A). Random weights; the point is which tokens can see which.
# Sequence: task tokens, then for each time step: observation tokens followed by one readout token.
rng = np.random.default_rng(0)
d = 4
layout = [("task", 2), ("obs t0", 3), ("readout t0", 1), ("obs t1", 3), ("readout t1", 1)]
kind, step = [], []
for name, n in layout:
    for _ in range(n):
        kind.append("task" if name == "task" else "readout" if name.startswith("readout") else "obs")
        step.append(-1 if name == "task" else int(name[-1]))
N = len(kind)
allowed = np.zeros((N, N), dtype=bool)   # allowed[i, j]: token i may look at token j
for i in range(N):
    for j in range(N):
        if kind[j] == "readout":
            allowed[i, j] = (i == j)                      # nobody attends to readout tokens (except themselves)
        elif kind[i] == "task":
            allowed[i, j] = kind[j] == "task"             # task tokens look at task tokens
        else:
            allowed[i, j] = kind[j] == "task" or step[j] <= step[i]   # observation and readout tokens: task + same or earlier steps
count = {}
labels = []
for k, s in zip(kind, step):
    key = (k, s); count[key] = count.get(key, 0) + 1
    labels.append(f"{k[0]}{s if s >= 0 else ''}.{count[key]}" if k != "readout" else f"r{s}")
print("token order:", " ".join(labels), "  (t = task, o = observation of step 0 or 1, r = readout)")
for lab, row in zip(labels, allowed.astype(int)):
    print(f"{lab:>5} sees", "".join(map(str, row)))

Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
def attend(x, mask):
    s = (x @ Wq) @ (x @ Wk).T / np.sqrt(d)
    s = np.where(mask, s, -1e9)
    w = np.exp(s - s.max(axis=1, keepdims=True))
    return (w / w.sum(axis=1, keepdims=True)) @ (x @ Wv)

x = rng.normal(size=(N, d))
base = attend(x, allowed)
# 1) change the first readout token's input: only that token's output may change
x2 = x.copy(); x2[labels.index("r0")] += 5
moved = np.abs(attend(x2, allowed) - base).max(axis=1) > 1e-9
print("change a readout token, outputs that move:", [labels[i] for i in np.flatnonzero(moved)])
# 2) change an observation token of time step 1: earlier tokens must not move
x3 = x.copy(); x3[labels.index("o1.1")] += 5
moved = np.abs(attend(x3, allowed) - base).max(axis=1) > 1e-9
print("change an observation at t1, outputs that move:", [labels[i] for i in np.flatnonzero(moved)])
# 3) drop a modality (mask out the task tokens, as for a dataset without language): the rest still runs
drop = allowed.copy(); drop[:, [i for i, k in enumerate(kind) if k == "task"]] = False
for i in [i for i, k in enumerate(kind) if k == "task"]: drop[i, i] = True
out = attend(x, drop)
print("without task tokens the sequence still produces", out.shape[0], "outputs; finite:", bool(np.isfinite(out).all()))

Output:

token order: t.1 t.2 o0.1 o0.2 o0.3 r0 o1.1 o1.2 o1.3 r1   (t = task, o = observation of step 0 or 1, r = readout)
  t.1 sees 1100000000
  t.2 sees 1100000000
 o0.1 sees 1111100000
 o0.2 sees 1111100000
 o0.3 sees 1111100000
   r0 sees 1111110000
 o1.1 sees 1111101110
 o1.2 sees 1111101110
 o1.3 sees 1111101110
   r1 sees 1111101111
change a readout token, outputs that move: ['r0']
change an observation at t1, outputs that move: ['o1.1', 'o1.2', 'o1.3', 'r1']
without task tokens the sequence still produces 10 outputs; finite: True

Random weights and a tiny sequence; it illustrates the mask, not the trained model.

Evidence in the paper
§III-A · Figure 2
Check your understanding: What does masking out the language tokens do to the other tokens?

Nothing is retrained: the other tokens simply do not attend to the missing ones, so a dataset without instructions can be processed by the same network.

3. Readout tokens: a place to read the answer

Intuition
A notepad that collects what the transformer understood, but that no other token ever reads.
The problem it solves
The action head needs a compact summary of the observations, and adding or changing heads must not disturb the rest of the model.
How it works
After the observation tokens of each time step the model inserts learned readout tokens. A readout token can attend to the task tokens and to the observation tokens before it, but no other token attends to it. The paper compares them to the classification token of BERT, a language model. The action head is applied to the readout embeddings only.
Example
Illustration, not from the paper A new action head for a two-armed robot reads the same readout tokens as the old one and adds nothing to the transformer’s computation.
Why the authors chose it
Because nothing depends on the readouts, new output heads can be attached later without changing any pretrained parameters.
Technical details
The same mask example above also checks this property: changing a readout token moves only its own output.
Evidence in the paper
§III-A · Figure 2 (bottom)
Check your understanding: Why must nothing attend to the readout tokens?

If other tokens read them, a new head or a changed readout would change everything downstream, and the pretrained weights would no longer fit.

4. The diffusion action head

Intuition
Start from random actions and clean them up in a few steps, so that the robot can pick one of several good actions instead of a muddy average.
The problem it solves
A regression head trained with mean squared error predicts the average of the demonstrations, which is poor when two different actions are both good, such as going left or right around an obstacle. Discretised actions can be decisive but lose precision.
How it works
A small head, a three-layer network with 256 hidden units, takes the transformer’s readout embedding and a noisy action chunk and predicts the noise to remove. Sampling starts from Gaussian noise and applies 20 denoising steps with a cosine noise schedule, as in a denoising diffusion probabilistic model (DDPM). Only the small head is run repeatedly; the transformer runs once. The head predicts a chunk of several consecutive actions.
Example
Illustration, not from the paper Two demonstrations go around a cup, one on each side. A diffusion head can output either path; a mean-squared-error head outputs the path through the cup.
Why the authors chose it
The authors report this head beat both alternatives in zero-shot and finetuning tests, and describe the policy as acting more decisively than the regression head and more precisely than the discrete one.
Technical details
Toy illustration The example uses a made-up two-action problem and the exact denoiser for it, so the 20-step sampling loop is real but nothing is trained. It shows the “hedging” of a mean-squared-error head and the two modes a diffusion sampler keeps.
Run the example: two equally good actions
Toy illustration

What each part does

  1. Make demonstrations that go left (−1) or right (+1).
  2. Show what a mean-squared-error head predicts.
  3. Run a 20-step diffusion sampler with the exact denoiser and count where the samples land.
import numpy as np

# Why a diffusion head instead of a mean-squared-error head? A toy with two equally good actions: go left (-1) or go right (+1).
# No network is trained: the denoiser below is the exact one for this two-bump distribution, so the sampling loop (20 steps, cosine schedule) is real
# and the data are made up. It illustrates the "hedging" the paper describes, not Octo's results.
rng = np.random.default_rng(0)
s = 0.05                                    # width of each bump
mu = np.array([-1.0, 1.0])
data = rng.choice(mu, 20000) + s * rng.normal(size=20000)

# a mean-squared-error head learns the average of the demonstrations
mse_prediction = data.mean()
print(f"mean-squared-error head: predicts {mse_prediction:+.3f} every time; fraction of predictions within 0.3 of a demonstrated action: "
      f"{float(np.mean(np.abs(mu - mse_prediction).min() < 0.3)):.0%}")

K = 20                                      # diffusion steps (Appendix D)
t = np.linspace(0, 1, K + 1)
off = 0.008
abar = np.cos((t + off) / (1 + off) * np.pi / 2) ** 2
abar = abar / abar[0]                       # cosine noise schedule
abar = np.clip(abar, 1e-4, 1.0)
beta = np.clip(1 - abar[1:] / abar[:-1], 0, 0.999)
alpha = 1 - beta

def x0_hat(xk, k):
    # exact posterior mean of the clean action for a two-bump distribution at noise level k
    ab = abar[k]
    var = ab * s * s + (1 - ab)
    logw = -0.5 * (xk[:, None] - np.sqrt(ab) * mu[None, :]) ** 2 / var
    w = np.exp(logw - logw.max(axis=1, keepdims=True)); w /= w.sum(axis=1, keepdims=True)
    gain = np.sqrt(ab) * s * s / var
    return (w * (mu[None, :] + gain * (xk[:, None] - np.sqrt(ab) * mu[None, :]))).sum(axis=1)

n = 5000
x = rng.normal(size=n)                      # start from pure noise
for k in range(K, 0, -1):
    ab_k, ab_prev = abar[k], abar[k - 1]
    eps = (x - np.sqrt(ab_k) * x0_hat(x, k)) / np.sqrt(1 - ab_k)          # the denoising network's output
    mean = (x - beta[k - 1] / np.sqrt(1 - ab_k) * eps) / np.sqrt(alpha[k - 1])
    var = beta[k - 1] * (1 - ab_prev) / (1 - ab_k)
    x = mean + (np.sqrt(var) * rng.normal(size=n) if k > 1 else 0)
near_left, near_right = float(np.mean(np.abs(x + 1) < 0.3)), float(np.mean(np.abs(x - 1) < 0.3))
print(f"diffusion head: {near_left:.0%} of samples near -1, {near_right:.0%} near +1, {1 - near_left - near_right:.0%} elsewhere")
print(f"mean of the samples {x.mean():+.3f} (the mean-squared-error answer), but only {float(np.mean(np.abs(x) < 0.3)):.1%} of them are near it")

Output:

mean-squared-error head: predicts +0.002 every time; fraction of predictions within 0.3 of a demonstrated action: 0%
diffusion head: 50% of samples near -1, 50% near +1, 0% elsewhere
mean of the samples -0.000 (the mean-squared-error answer), but only 0.0% of them are near it

Made-up data; no learned network. It illustrates the claim, it is not Octo’s result.

Evidence in the paper
§III-C · Appendix D · Appendix E (“Things that did not work”) · Equation 1
Check your understanding: What does the diffusion head model that a mean-squared-error head cannot?

Several different good actions at once. The regression head collapses them to their average, which may be a bad action; the diffusion sampler returns one of the good ones.

5. Adapting to new inputs and outputs

Intuition
Plug in a new sensor or a new set of motors without rebuilding the brain.
The problem it solves
Real robots differ in cameras, sensors and action formats, and a pretrained model should not need to be rebuilt for each.
How it works
When finetuning, the pretrained transformer weights are kept. A new input needs only new position embeddings and a lightweight encoder; a new output needs a new head. The paper’s examples are force and torque readings for an insertion task, joint-position control for a pick-up task, and a new 14-number action head for a two-armed robot. All weights, including the transformer, are updated during finetuning, which the authors found better than freezing parts.
Example
Illustration, not from the paper A new force-torque sensor gets its own small encoder whose tokens join the sequence; the rest of the model is unchanged.
Why the authors chose it
It is what the authors mean by a generalist policy that can be applied to setups not seen in pretraining.
Technical details
The two-armed robot (ALOHA) example predicts a chunk of 64 actions in training and executes 12 before planning again. The recipe is the same for every domain: about 100 trajectories, 50 thousand steps, a cosine learning-rate decay.
Evidence in the paper
§III-A · §III-C · Appendix F.C
Check your understanding: Is the pretrained transformer frozen during finetuning?

No. It is kept as the starting point and trained along with the new parts; the authors found updating everything better than freezing subsets.

Part C

Understand the data and the training

The training idea is simple imitation on a very large pooled dataset. What needs care is the data: 25 datasets that differ in robots, cameras, labels and size have to be balanced into one stream.

The data: 25 datasets from Open X-Embodiment

Intuition
Pool many labs’ demonstrations into one training stream.
The problem it solves
No single robot dataset covers enough scenes, objects and skills to train a generalist.
How it works
The authors take 25 of the datasets of Open X-Embodiment, a collection of about 1.5 million episodes, and keep 800 thousand episodes. An earlier model (RT-X) used 350 thousand. Each dataset contributes to every batch in proportion to its size, with a few manual changes. The three largest datasets (Fractal, Kuka and Bridge) are 17% each.
Example
Illustration, not from the paper A batch of 2,048 examples contains about 350 examples from each of Fractal, Kuka and Bridge and a handful from the smallest datasets.
Why the authors chose it
The authors argue the wide mixture is what makes the policy transfer across robots, and test it in an ablation on one robot.
Technical details

Our calculation By our arithmetic the top three datasets make up 51.0% of every batch, the top five 66.1% and the top ten 84.8%; eight datasets are below 1% (2.55% together). Bridge, the data of the WidowX robot used in the ablations, is 17%.

Run the example: who dominates a batch?
Our calculation

What each part does

  1. Type in the shares of Table III.
  2. Add them up and compute the share of the largest datasets.
  3. Add the wrist-camera and language shares quoted in the discussion.
# The pretraining mixture (Appendix C, Table III): the share of each batch that comes from each of the 25 datasets.
share = {"Fractal": 17.0, "Kuka": 17.0, "Bridge": 17.0, "BC-Z": 9.1, "Stanford Hydra": 6.0, "Language Table": 5.9, "Taco Play": 3.6, "Furniture Bench": 3.3,
         "UTAustin Mutex": 3.0, "Austin Sailor": 2.9, "Roboturk": 2.8, "Toto": 2.4, "Austin Sirius": 2.3, "Berkeley Autolab UR5": 1.5, "IAMLab CMU Pickup Insert": 1.2,
         "Viola": 1.2, "Berkeley Fanuc": 1.0, "NYU Franka Play": 0.9, "UCSD Kitchen": 0.05, "Jaco Play": 0.6, "Berkeley Cable Routing": 0.3, "Austin Buds": 0.3,
         "CMU Stretch": 0.2, "NYU Door Opening": 0.1, "DLR EDAN": 0.1}      # UCSD Kitchen is printed as "<0.1%"; we use 0.05
ordered = sorted(share.values(), reverse=True)
print("datasets listed:", len(share), "| shares add up to", round(sum(share.values()), 2), "%")
for k in (1, 3, 5, 10):
    print(f"top {k:2d} datasets: {sum(ordered[:k]):5.1f}% of every batch")
print("datasets with at least 1% :", sum(v >= 1 for v in ordered))
print("datasets below 1%         :", sum(v < 1 for v in ordered), f"(together {sum(v for v in ordered if v < 1):.2f}%)")
print(f"Bridge (the WidowX data used in the ablations): {share['Bridge']}% of the mixture")

# text statistics of the mixture (Section V)
print()
print("data with wrist-camera images: 27% | data with language annotations: 56% (Section V)")
print("so about", round(100 - 27), "% of the data has no wrist camera and about", round(100 - 56), "% has no instruction (goal images are used for them)")

Output:

datasets listed: 25 | shares add up to 99.75 %
top  1 datasets:  17.0% of every batch
top  3 datasets:  51.0% of every batch
top  5 datasets:  66.1% of every batch
top 10 datasets:  84.8% of every batch
datasets with at least 1% : 17
datasets below 1%         : 8 (together 2.55%)
Bridge (the WidowX data used in the ablations): 17.0% of the mixture

data with wrist-camera images: 27% | data with language annotations: 56% (Section V)
so about 73 % of the data has no wrist camera and about 44 % has no instruction (goal images are used for them)

Arithmetic on Table III; “<0.1%” is entered as 0.05.

Evidence in the paper
§II · §III-B · Figure 3 · Appendix C (Table III)
Check your understanding: Why does it matter that three datasets make up half of every batch?

A “wide” mixture of 25 datasets is in practice dominated by three, so the policy sees the robots and scenes of those much more than the long tail of the others.

Curating and aligning the data

Intuition
Make datasets that were recorded differently look alike, without throwing away what makes them useful.
The problem it solves
The datasets differ in cameras, action formats and labels: some have a wrist camera or language instructions and some do not.
How it works
The authors drop datasets with no images or that do not use delta end-effector control, and those that are too repetitive, of low resolution or too niche. They sort the rest into “more diverse” and “less diverse” and double the weight of the more diverse ones; a few repetitive datasets are down-weighted. Missing camera channels are filled with zeros. Gripper commands are aligned so +1 means open and 0 means closed.
Example
Illustration, not from the paper A dataset with only one camera has the wrist channel padded with zeros and is masked out of the attention, so the model treats it as “no wrist camera”.
Why the authors chose it
It lets a single model train on all of them. The authors say future work should study the quality of the mixture more thoroughly.
Technical details
Only 27% of the data has wrist-camera images and 56% has language annotations. For gripper actions, the authors tried a relative representation (+1 only at the moment of opening, 0.5 otherwise) and found it grasped slightly more often but retried less after a failed grasp, so they kept the absolute one. They also found that a very large shuffle buffer (up to 500 thousand frames, with at most 100 steps taken per trajectory) was crucial for zero-shot performance.
Evidence in the paper
§III-B · §V · Appendix E
Check your understanding: What happens to the model when a dataset has no language instructions?

The language tokens are masked out and the dataset is trained with goal images only; this is the 44% of the data that has no instruction.

The training objective and the tricks around it

Intuition
Learn to remove noise from demonstrated actions, and make the task description optional.
The problem it solves
The model must work whether the user gives a sentence, a goal image or neither, and must learn from data without any language.
How it works
The action head is trained with the standard diffusion objective: noise is added to the demonstrated actions and the head learns to recover them. The task input is varied during training: for each example the instruction or the goal image is randomly zeroed out, and for datasets without language the goal image is always used. The goal image is chosen by hindsight relabelling: a state sampled uniformly from later in the same trajectory. Common image augmentations are applied.
Example
Illustration, not from the paper For a trajectory of 100 steps, the frame at step 70 can serve as the goal image for the example at step 20.
Why the authors chose it
Goal images need no language labels, so the model can learn from the large share of data that lacks them.
Technical details
Two frames of observation history are used; the authors found diminishing gains beyond the first extra frame. The same objective is used for finetuning. The paper compares predicting actions with the diffusion head against mean squared error and discretised cross-entropy in Part D.
Evidence in the paper
§III-C · §III-D · Appendix E
Check your understanding: Why use goal images chosen from later in the same trajectory?

That state is by construction reachable from the current one, and it comes for free from the recording, with no human labelling.

Sizes, cost and the finetuning recipe

Intuition
A model small enough that a lab can finetune it on one graphics card.
The problem it solves
A generalist policy is only useful to others if they can adapt it with the hardware they have.
How it works
Two sizes are released: Octo-Small (27 million parameters) and Octo-Base (93 million). Octo-Base was trained for 300 thousand steps with a batch size of 2,048 on a tensor processing unit (TPU) v4-128 pod in 14 hours. Finetuning the same model on one NVIDIA A5000 graphics card with 24 GB takes about 5 hours. Training uses AdamW with a reciprocal-square-root learning-rate schedule, a peak learning rate of 3e-4, 2,000 warm-up steps, weight decay 0.1 and gradient clipping at 1. A third size, Octo-Tiny (10 million parameters), appears in the scaling figure.
Example
Illustration, not from the paper A lab with 100 demonstrations and one consumer GPU can finetune Octo-Base overnight.
Why the authors chose it
The authors stress usability: the checkpoints, the finetuning scripts in JAX, the pretraining pipeline and the data loaders are all released.
Technical details
Table V, as printed.
ModelLayersHidden sizeMLP sizeHeadsParameters
Octo-Small123841,536627 M
Octo-Base127683,0721293 M

Our calculation Our estimate of the transformer blocks alone is 21.2 and 84.9 million parameters, so the remaining 5.8 and 8.1 million are the image stems, embeddings, readouts and the head. The frozen T5-base language encoder has 111 million parameters and is not part of the stated 27 and 93 million, so the whole Octo-Base model needs about 204 million. Pretraining processed about 12,190 samples a second.

Run the example: size, language encoder and cost
Our calculation

What each part does

  1. Estimate the weights in the transformer blocks from the sizes in Table V.
  2. Add the frozen language encoder and compare with RT-1-X.
  3. Work out the pretraining throughput and the finetuning speed.
# Sizes and cost, from Appendix D (Tables IV and V) and Section III-D.
rows = {"Octo-Small": (12, 384, 1536, 27e6), "Octo-Base": (12, 768, 3072, 93e6)}
print("transformer blocks only (attention + MLP weights, no embeddings, norms or heads):")
for name, (layers, d, mlp, stated) in rows.items():
    per_layer = 4 * d * d + 2 * d * mlp
    blocks = layers * per_layer
    print(f"  {name}: {per_layer / 1e6:.2f} M per layer x {layers} = {blocks / 1e6:5.1f} M of the stated {stated / 1e6:.0f} M ({blocks / stated:.0%}); the rest, {(stated - blocks) / 1e6:.1f} M, is the CNN patch stems, embeddings, readouts and the action head")

t5 = 111e6
print()
print(f"frozen T5-base language encoder: {t5 / 1e6:.0f} M parameters, not part of the stated 27 M and 93 M")
for name, (_, _, _, stated) in rows.items():
    print(f"  {name} plus the language encoder: about {(stated + t5) / 1e6:.0f} M")
print(f"  RT-1-X is listed at 35 M; Octo-Base alone is {93 / 35:.1f} times that, with the language encoder {(93 + 111) / 35:.1f} times")

steps, batch, hours = 300_000, 2048, 14
samples = steps * batch
print()
print(f"pretraining: {steps:,} steps x batch {batch} = {samples / 1e6:.1f} M samples in {hours} h -> {samples / (hours * 3600):,.0f} samples per second")
print(f"finetuning: 50,000 steps in about 5 h on one A5000 -> {50_000 / (5 * 3600):.1f} steps per second (the batch size is not stated)")
print()
for name, tokens in (("third-person camera, 256 x 256 (16 x 16 patches)", 256), ("wrist camera, 128 x 128 (16 x 16 patches)", 64), ("language instruction (T5-base embeddings)", 16)):
    print(f"  {name:50s} {tokens:3d} tokens")
per_step = 256 + 64 + 16
print(f"with two cameras and an instruction: {per_step} input tokens per time step; with 2 frames of history about {2 * (256 + 64) + 16} plus readout tokens")

Output:

transformer blocks only (attention + MLP weights, no embeddings, norms or heads):
  Octo-Small: 1.77 M per layer x 12 =  21.2 M of the stated 27 M (79%); the rest, 5.8 M, is the CNN patch stems, embeddings, readouts and the action head
  Octo-Base: 7.08 M per layer x 12 =  84.9 M of the stated 93 M (91%); the rest, 8.1 M, is the CNN patch stems, embeddings, readouts and the action head

frozen T5-base language encoder: 111 M parameters, not part of the stated 27 M and 93 M
  Octo-Small plus the language encoder: about 138 M
  Octo-Base plus the language encoder: about 204 M
  RT-1-X is listed at 35 M; Octo-Base alone is 2.7 times that, with the language encoder 5.8 times

pretraining: 300,000 steps x batch 2048 = 614.4 M samples in 14 h -> 12,190 samples per second
finetuning: 50,000 steps in about 5 h on one A5000 -> 2.8 steps per second (the batch size is not stated)

  third-person camera, 256 x 256 (16 x 16 patches)   256 tokens
  wrist camera, 128 x 128 (16 x 16 patches)           64 tokens
  language instruction (T5-base embeddings)           16 tokens
with two cameras and an instruction: 336 input tokens per time step; with 2 frames of history about 656 plus readout tokens

Arithmetic on Tables IV and V and the text of Section III-D; the block formula ignores embeddings and norms.

Evidence in the paper
§III-D · §III-E · Appendix D (Tables IV and V)
Check your understanding: Which cost number would you ask for to plan a finetuning run, and is it given?

The batch size and steps per second of finetuning. The paper gives 50 thousand steps and about 5 hours but not the finetuning batch size.

Part D

Understand the experiments

There are three questions: can Octo control robots out of the box, does it finetune well to new setups, and which design choices matter. This part explains each test, what the tables say and what they cannot show. Our re-calculations are in the folded examples.

The set-up: nine robot setups, baselines and trials

How it works
Three setups test zero-shot use with tasks taken from the pretraining data: a WidowX arm (the Bridge setup), a UR5 and the RT-1 robot, each with two language tasks and 10 trials per task. Six setups test finetuning: CMU Baking, Stanford Coffee, Berkeley Insertion (new force-torque input), Berkeley Pick-Up (new joint-position action space), Berkeley Coke (a new robot, a ViperX) and Berkeley Bimanual (a two-armed robot). Zero-shot comparisons are with RT-1-X (35 million parameters) and RT-2-X (55 billion, a vision-language model finetuned on Open X-Embodiment). For finetuning there are two baselines. One is a from-scratch policy with a ResNet encoder, language conditioning by FiLM and a small transformer with a diffusion head, 28 million parameters, because a ViT trained from scratch overfitted quickly. The other is a ViT-B initialised from VC-1, a visual representation trained on 4,000 hours of egocentric video and ImageNet, with a mean-squared-error action decoder.
Technical details

Our calculation The trial counts are small: with 10 trials a success rate near 50% has a standard error of about 16 points, with 20 trials 11 and with 40 trials 8. An exact test calls two rates different only when they are at least 50, 35 and 25 points apart respectively.

Run the example: how much a success rate can move by chance
Our calculation

What each part does

  1. Take the trial counts used in the paper: 10, 20 and 40.
  2. Compute the standard error of a rate near 50%.
  3. Find the smallest gap an exact two-sided test treats as different.
import math
# How much can a success rate move by chance? Standard error and the smallest gap an exact test can call different, for the trial counts used in the paper.
def se(p, n): return 100 * math.sqrt(p * (1 - p) / n)
def p_two_sided(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)

print("trials | where in the paper                              | standard error at 50% | smallest detectable gap (baseline at 50%)")
for n, where in ((10, "per task in zero-shot tests and model scaling"), (20, "per finetuning domain; Table VII per row"), (40, "ablations (Tables II and VI)")):
    base = n // 2
    gap = next((100 * (k - base) / n for k in range(base, n + 1) if p_two_sided(k, n, base, n) < 0.05), None)
    print(f"{n:6d} | {where:47s} | {se(0.5, n):19.1f} pts | {gap:.0f} points")
print()
print("what that means for headline gaps (Octo against a baseline, same trial counts):")
for name, a, b, n in (("Octo-Small vs ResNet-50 + Transformer, WidowX", 83, 70, 40), ("Octo vs ResNet+Transformer scratch, Stanford Coffee", 75, 45, 20), ("Octo vs ResNet+Transformer scratch, CMU Baking", 50, 25, 20)):
    ka, kb = round(a * n / 100), round(b * n / 100)
    print(f"  {name:55s} {a}% vs {b}% (n={n}): p = {p_two_sided(ka, n, kb, n):.3f}")

Output:

trials | where in the paper                              | standard error at 50% | smallest detectable gap (baseline at 50%)
    10 | per task in zero-shot tests and model scaling   |                15.8 pts | 50 points
    20 | per finetuning domain; Table VII per row        |                11.2 pts | 35 points
    40 | ablations (Tables II and VI)                    |                 7.9 pts | 25 points

what that means for headline gaps (Octo against a baseline, same trial counts):
  Octo-Small vs ResNet-50 + Transformer, WidowX           83% vs 70% (n=40): p = 0.293
  Octo vs ResNet+Transformer scratch, Stanford Coffee     75% vs 45% (n=20): p = 0.105
  Octo vs ResNet+Transformer scratch, CMU Baking          50% vs 25% (n=20): p = 0.191

Our calculation, assuming independent trials; each experiment is a single run.

Evidence in the paper
§IV · Figure 4 · Appendix F
Check your understanding: Why is the from-scratch baseline not simply Octo’s own architecture without pretraining?

The authors found that the large transformer overfitted on a few hundred demonstrations, so they used a ResNet-based policy for the from-scratch comparison; the comparison therefore changes the architecture as well as the pretraining.

Controlling robots out of the box

How it works
On tasks drawn from the pretraining datasets, with new object positions, lighting, backgrounds and distractors, the authors report that Octo has a 29% higher success rate than RT-1-X on average, and performs similarly to RT-2-X on the WidowX and RT-1 robot setups. On the WidowX, conditioning on a goal image instead of a sentence gave 25% higher success. A finer analysis on the WidowX (Table VII) finds high success on in-distribution tasks and novel objects, lower on a new scene and very low on new skills such as flipping a cup (10%) or putting a block in a slot (0%).
Technical details
Table VII, Octo-Small on the WidowX, as printed; 10 trials per task.
Generalisation typeTaskSuccess (%)Average (%)
In-distributionput carrot on plate; put eggplant in pot80; 9085
Novel objectsput bread on plate; put spoon on glove70; 9080
Novel environmentput mushroom in pot; put spoon on cloth20; 6040
Novel skillflip cup on its side; put block in slot10; 05

Our interpretation The zero-shot comparison of Figure 5 is a bar chart with no values printed; the RT-2-X numbers for the WidowX are copied from another paper, and for the RT-1 robot the evaluation was run by the RT-2-X authors. The two “novel object” tasks are the goal-image tasks, so Table VII compares language and goal-image tasks (see Part F).

Evidence in the paper
§IV-A · Figure 5 · Table VII · Appendix F.A
Check your understanding: What does “zero-shot” mean here, given that the tasks come from the pretraining data?

No finetuning is done, but the robots and task types appear in the pretraining datasets; the model still has to handle new object positions, lighting and distractors. It does not mean unseen tasks.

Finetuning to new robots, sensors and action spaces (Table I)

How it works
Each of the six setups uses about 100 demonstrations and the same finetuning hyperparameters, and Octo is compared with the two baselines. Octo averages 72%, against 20% for the from-scratch policy and 15% for VC-1, the best baseline being 52 points behind. It reaches 100% on the Berkeley Coke task with a new robot, 70% on insertion with force-torque input and 60% on pick-up with joint-position control.
Technical details
Table I, success rate in percent, as printed. The caption says 20 trials per domain.
MethodInsertionCoffeeBakingPick-UpCokeBimanualAverage
ResNet+Transformer Scratch1045250202020
VC-150300105015
Octo707550601008072

Our calculation The averages of the six printed values are 15.83 (VC-1) and 72.5 (Octo), printed as 15 and 72, which is truncation; the ablation tables round half up. Octo beats the scratch policy clearly in total, but the gaps in single domains such as Stanford Coffee (15/20 against 9/20) and CMU Baking (10/20 against 5/20) are within chance.

Run the example: Table I: averages, trial counts and chance
Our calculation

What each part does

  1. Type in Table I and recompute the averages.
  2. Compare rounding with truncation and the difference in points with the text.
  3. Run an exact test per domain and pool the trials, with 10 and with 20 trials for Bimanual.
import math
# Table I (finetuning), success rates in percent, as printed. Columns: Berkeley Insertion, Stanford Coffee, CMU Baking, Berkeley Pick-Up, Berkeley Coke, Berkeley Bimanual.
domains = ["Insertion", "Coffee", "Baking", "Pick-Up", "Coke", "Bimanual"]
rows = {"ResNet+Transformer Scratch": ([10, 45, 25, 0, 20, 20], 20), "VC-1": ([5, 0, 30, 0, 10, 50], 15), "Octo": ([70, 75, 50, 60, 100, 80], 72)}
print("average of the six domains, recomputed, against the printed average")
for name, (v, printed) in rows.items():
    m = sum(v) / 6
    print(f"  {name:28s} mean {m:5.2f}  rounded {int(m + 0.5):3d}  truncated {int(m):3d}  printed {printed}")
print("  Table II uses rounding half up (82.5 -> 83, 42.5 -> 43, 17.5 -> 18); Table I matches truncation, not rounding")
print(f"  Octo minus the best baseline: {72 - 20} points (the text says 'by 52%'); relative: {72 / 20:.1f} times")

def p_two_sided(a, n1, b, n2):
    # exact two-sided Fisher test for success counts a of n1 and b of n2
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)

print()
print("trials: the caption says 20 per domain; Appendix F says 10 for Bimanual. Counts below use 20 for five domains and show Bimanual both ways.")
for i, dom in enumerate(domains):
    for n in ((20,) if dom != "Bimanual" else (10, 20)):
        o, s = round(rows["Octo"][0][i] * n / 100), round(rows["ResNet+Transformer Scratch"][0][i] * n / 100)
        print(f"  {dom:9s} n={n:2d}: Octo {o:2d}/{n}, scratch {s:2d}/{n}, exact p = {p_two_sided(o, n, s, n):.4f}")

print()
for n_b in (10, 20):
    tot = 5 * 20 + n_b
    pooled = {name: sum(round(v[i] * (n_b if i == 5 else 20) / 100) for i in range(6)) for name, (v, _) in rows.items()}
    print(f"pooled over all trials (Bimanual n={n_b}, {tot} trials): " + ", ".join(f"{k} {c / tot:5.1%}" for k, c in pooled.items()))

Output:

average of the six domains, recomputed, against the printed average
  ResNet+Transformer Scratch   mean 20.00  rounded  20  truncated  20  printed 20
  VC-1                         mean 15.83  rounded  16  truncated  15  printed 15
  Octo                         mean 72.50  rounded  73  truncated  72  printed 72
  Table II uses rounding half up (82.5 -> 83, 42.5 -> 43, 17.5 -> 18); Table I matches truncation, not rounding
  Octo minus the best baseline: 52 points (the text says 'by 52%'); relative: 3.6 times

trials: the caption says 20 per domain; Appendix F says 10 for Bimanual. Counts below use 20 for five domains and show Bimanual both ways.
  Insertion n=20: Octo 14/20, scratch  2/20, exact p = 0.0002
  Coffee    n=20: Octo 15/20, scratch  9/20, exact p = 0.1053
  Baking    n=20: Octo 10/20, scratch  5/20, exact p = 0.1908
  Pick-Up   n=20: Octo 12/20, scratch  0/20, exact p = 0.0000
  Coke      n=20: Octo 20/20, scratch  4/20, exact p = 0.0000
  Bimanual  n=10: Octo  8/10, scratch  2/10, exact p = 0.0230
  Bimanual  n=20: Octo 16/20, scratch  4/20, exact p = 0.0004

pooled over all trials (Bimanual n=10, 110 trials): ResNet+Transformer Scratch 20.0%, VC-1 12.7%, Octo 71.8%
pooled over all trials (Bimanual n=20, 120 trials): ResNet+Transformer Scratch 20.0%, VC-1 15.8%, Octo 72.5%

Arithmetic on the printed table; the trial counts are those the paper states.

Evidence in the paper
§IV-B · Table I · Appendix F.C
Check your understanding: Why are the baselines in Table I a fair test of “pretraining helps”, and where are they not?

They show that the Octo recipe beats strong, commonly used alternatives on the same 100 demonstrations. They are not a clean test of pretraining alone, because the scratch policy has a different architecture and VC-1 a different loss.

Which design choices matter (Tables II and VI, Figure 6)

How it works
All ablations are on the WidowX setup with Octo-Small: two language-conditioned tasks and two goal-image tasks, 10 trials each. The chosen design scores 83%. Replacing the diffusion head with mean squared error gives 35% and with discretised actions 18%. A ResNet-50 encoder with a transformer scores 70%. Training on the 11 datasets of RT-X gives 60%, and on Bridge alone 43%. A scaling figure (10, 27 and 93 million parameters, 10 trials of one task per robot) shows zero-shot success improving with size.
Technical details
Table VI, success rate in percent, as printed; the first two tasks are language-conditioned and the last two goal-conditioned.
VariantCarrotEggplantBreadSpoonAverage
Octo-Small (ours)8090709083
RT-X data mix8080404060
Single robot dataset (Bridge)2070602043
Discretized action prediction020104018
Continuous action prediction (mean squared error, MSE)703004035
ResNet-50 + Transformer80601004070

Our calculation Against Octo-Small’s 33 successes in 40 trials, the diffusion-head, Bridge-only and discretised comparisons are far beyond chance. The comparison with the 11-dataset mix is borderline (24/40, p of 0.047), and the ResNet-50 comparison (28/40, p of 0.293) is within chance.

Run the example: Tables II, VI and VII: counts and chance
Our calculation

What each part does

  1. Type in Table VI and recompute the averages.
  2. Convert them to counts out of 40 trials and run an exact test against Octo-Small.
  3. Compare the language-conditioned and goal-image tasks.
import math
# Tables II and VI (ablations on the WidowX setup), per-task success in percent, as printed. 10 trials per task, 4 tasks -> 40 trials ("averaged over 40 trials").
tasks = ["carrot on plate (language)", "eggplant in pot (language)", "bread on plate (goal image)", "spoon on glove (goal image)"]
rows = {
    "Octo-Small (ours)":                      ([80, 90, 70, 90], 83),
    "RT-X data mix (11 datasets)":            ([80, 80, 40, 40], 60),
    "Single robot dataset (Bridge)":          ([20, 70, 60, 20], 43),
    "Discretized action prediction":          ([0, 20, 10, 40], 18),
    "Continuous action prediction (MSE)":     ([70, 30, 0, 40], 35),
    "ResNet-50 + Transformer":                ([80, 60, 100, 40], 70),
}
print("average of the four tasks, recomputed:")
for name, (v, printed) in rows.items():
    m = sum(v) / 4
    print(f"  {name:36s} mean {m:5.2f}  rounded half up {int(m + 0.5):3d}  printed {printed}")

def p_two_sided(a, n1, b, n2):
    m, N = a + b, n1 + n2
    pr = lambda x: math.comb(m, x) * math.comb(N - m, n1 - x) / math.comb(N, n1)
    p0 = pr(a)
    return sum(pr(x) for x in range(max(0, m - n2), min(m, n1) + 1) if pr(x) <= p0 + 1e-12)

n = 40
base = round(sum(rows["Octo-Small (ours)"][0]) * n / 400)
print()
print(f"Octo-Small: {base} successes out of {n}. Exact two-sided test against each ablation (40 trials each):")
for name, (v, _) in list(rows.items())[1:]:
    k = round(sum(v) * n / 400)
    print(f"  {name:36s} {k:2d}/{n}  gap {100 * (base - k) / n:4.1f} points  p = {p_two_sided(base, n, k, n):.3f}")

print()
print("Table VII (zero-shot generalisation), per task, 10 trials each: the two 'novel object' tasks are the goal-image tasks above")
for kind, vals in (("in-distribution (language)", [80, 90]), ("novel objects (goal image)", [70, 90]), ("novel environment", [20, 60]), ("novel skill", [10, 0])):
    print(f"  {kind:28s} {vals} average {sum(vals) / 2:.0f}")
lang, goal = sum(rows["Octo-Small (ours)"][0][:2]) / 2, sum(rows["Octo-Small (ours)"][0][2:]) / 2
print(f"Octo-Small: language-conditioned tasks {lang:.0f}%, goal-image tasks {goal:.0f}% (the text says goal images gave 25% higher success than language on the WidowX tasks)")

Output:

average of the four tasks, recomputed:
  Octo-Small (ours)                    mean 82.50  rounded half up  83  printed 83
  RT-X data mix (11 datasets)          mean 60.00  rounded half up  60  printed 60
  Single robot dataset (Bridge)        mean 42.50  rounded half up  43  printed 43
  Discretized action prediction        mean 17.50  rounded half up  18  printed 18
  Continuous action prediction (MSE)   mean 35.00  rounded half up  35  printed 35
  ResNet-50 + Transformer              mean 70.00  rounded half up  70  printed 70

Octo-Small: 33 successes out of 40. Exact two-sided test against each ablation (40 trials each):
  RT-X data mix (11 datasets)          24/40  gap 22.5 points  p = 0.047
  Single robot dataset (Bridge)        17/40  gap 40.0 points  p = 0.000
  Discretized action prediction         7/40  gap 65.0 points  p = 0.000
  Continuous action prediction (MSE)   14/40  gap 47.5 points  p = 0.000
  ResNet-50 + Transformer              28/40  gap 12.5 points  p = 0.293

Table VII (zero-shot generalisation), per task, 10 trials each: the two 'novel object' tasks are the goal-image tasks above
  in-distribution (language)   [80, 90] average 85
  novel objects (goal image)   [70, 90] average 80
  novel environment            [20, 60] average 40
  novel skill                  [10, 0] average 5
Octo-Small: language-conditioned tasks 85%, goal-image tasks 80% (the text says goal images gave 25% higher success than language on the WidowX tasks)

Arithmetic on the printed table; one run per row.

Evidence in the paper
§IV-C · Tables II and VI · Figure 6 · Appendix F.B
Check your understanding: Which ablation results are the most trustworthy, and why?

The very large ones: the discretised and mean-squared-error heads and the single-dataset data. A 13-point gap, such as the ViT against ResNet encoder, is about what chance could produce with 40 trials each.

What worked and what did not (Appendix E)

How it works
Appendix E lists observations that did not make it into the main tables. Helpful: one frame of history, action chunking (and no extra gain from temporal ensembling), 16 by 16 patches over 32 by 32, and a much larger shuffle buffer. Not (yet) helpful: mean-squared-error heads, which gave slow “hedging” policies; discretised action heads, which were decisive but imprecise; ResNet encoders on the large mixture (though better than the ViT on 100 demonstrations); ImageNet-pretrained ResNets for zero-shot use; a relative gripper representation; proprioceptive inputs, which seemed generally worse, possibly through causal confusion; and finetuning or enlarging the T5 language encoder, where the frozen base model gave the best language-conditioned policies.
Technical details
The authors present these as observations from their experiments, without tables. For readers building a similar system they are a checklist of design choices that were tried.
Evidence in the paper
Appendix E
Check your understanding: Proprioceptive inputs seemed worse in pretraining, yet two finetuning tasks use them. How can both be true?

Pretraining omits proprioception, and the authors suspect a causal confusion between the state and the future actions. The finetuning tasks that need force-torque or joint readings add them as new inputs, and the paper does not test whether those tasks would do better without them.

What the results show

  • A 27 or 93 million-parameter pretrained transformer can be finetuned in about five hours on one GPU, with about 100 demonstrations, to new sensors, action formats and robots, and clearly outperforms the from-scratch and VC-1 baselines in total.
  • On the WidowX ablations, a diffusion head and a wide data mix are far better than mean-squared-error and discretised heads or a single dataset.
  • Zero-shot success in pretraining-like settings is above RT-1-X and similar to RT-2-X according to the paper’s chart.

What they do not show

  • That the smaller gaps (encoder, data mix, single finetuning domains) are more than chance: one run per entry with 10 to 40 trials.
  • That pretraining alone explains the finetuning gain, since the baselines differ in architecture and loss.
  • That the model follows language well or uses wrist cameras well; the authors say it does not.
  • Anything about unseen tasks or skills: new skills reach 5% and the zero-shot tasks come from the pretraining data.

Part E · optional

Engineering deep dive

The denoising equation, the hyperparameters, the mixture, the evaluation setups and the code interface, for readers who want to rebuild or audit the work. Everything is collapsed; Parts A to D do not depend on it.

The denoising step

Reported by the paper Sampling an action chunk starts from Gaussian noise xK and applies K steps of a learned denoiser εθ, conditioned on the transformer’s readout embedding e and the step index k (Equation 1):

xk−1=α(xk−γ εθ(xk, e, k)+N(0, σ2 I))x^{k-1}=\alpha (x^{k}-\gamma \,\varepsilon _{\theta }(x^{k},\,e,\,k)+N(0,\,\sigma ^{2}\,I))

α, γ and σ come from the noise schedule: the cosine schedule of an earlier diffusion paper, with K = 20 steps. Training adds Gaussian noise to the demonstrated actions and trains εθ to recover the noise (the standard DDPM objective). Flow matching, used by π0, GR00T N1, SmolVLA and AsyncVLA, replaces this noise-prediction target with a direction along a path; see the Flow Matching guide.

Hyperparameters (Appendix D)
As printed in Tables IV and V and Sections III-C and III-D.
SettingValue
OptimiserAdamW, weight decay 0.1, gradient clipping 1.0
Learning rate3e-4, reciprocal square root decay, 2,000 warm-up steps
Batch size, steps (pretraining)2,048; 300,000 (ViT-B size)
Third-person imagerandom crop, resize to 256 × 256, colour jitter, pixels in [−1, 1]
Wrist imagecolour jitter, resize to 128 × 128, no random crop
Patch size, tokens16 × 16; 256 third-person, 64 wrist; 16 language
Diffusion head3-layer MLP, hidden 256, residual connections and layer norm; DDPM, cosine schedule, 20 steps
History, goal relabelling2 frames; goal image sampled uniformly from the future of the trajectory
Finetuningabout 100 trajectories, 50,000 steps, cosine decay with linear warm-up
The pretraining mixture (Table III)
As printed. Our count of the rows is 25 and the shares add up to about 99.8%.
DatasetShare
Fractal17.0%
Kuka17.0%
Bridge17.0%
BC-Z9.1%
Stanford Hydra6.0%
Language Table5.9%
Taco Play3.6%
Furniture Bench3.3%
UTAustin Mutex3.0%
Austin Sailor2.9%
Roboturk2.8%
Toto2.4%
Austin Sirius2.3%
Berkeley Autolab UR51.5%
IAMLab CMU Pickup Insert1.2%
Viola1.2%
Berkeley Fanuc1.0%
NYU Franka Play0.9%
Jaco Play0.6%
Berkeley Cable Routing0.3%
Austin Buds0.3%
CMU Stretch0.2%
NYU Door Opening0.1%
DLR EDAN0.1%
UCSD Kitchen< 0.1%
The evaluation setups (Appendix F)
  • Reported by the paper WidowX BridgeV2: a 6-degree-of-freedom Trossen WidowX 250, one third-person camera, end-effector position deltas; tasks “place the carrot on plate” and “put the eggplant in the pot”; 10 trials per task.
  • Reported by the paper UR5 and RT-1 robot: UR5 tasks are moving a toy tiger between bowls and wiping a table; the RT-1 robot picks cans, apples and chip bags and opens and closes drawers; both with one third-person camera and 10 trials per task, and a re-assembled or moved setup.
  • Reported by the paper CMU Baking: 120 demonstrations, end-effector deltas at 15 Hz, 20 trials (4 new toy breads, 5 positions). Stanford Coffee: 118 demonstrations, 10 Hz, 20 episodes (4 pod colours).
  • Reported by the paper Berkeley Insertion: 100 demonstrations from the FMB dataset, a 1.5 mm tolerance peg, force-torque input, actions at 5 Hz, 20 trials. Pick-Up: 100 pickups, joint-position control, wrist camera and joint readings, 20 trials.
  • Reported by the paper Berkeley Coke: a ViperX robot unseen in pretraining, 115 demonstrations, third-person and wrist cameras, 5 Hz. Berkeley Bimanual: an ALOHA robot, new 14-number action head, chunk of 64, 12 actions executed before replanning, 10 trials.
Running a released checkpoint

Reported by the paper Appendix B shows about nine lines of Python: load the pretrained model from a hub path, print its input-output specification, build an observation with a primary image, create a task from a text such as “pick up the fork”, and call a sampling method with a random key to get actions. Octo-Small and Octo-Base checkpoints, finetuning scripts and the pretraining pipeline, all in JAX, and standalone data loaders for JAX and PyTorch are released.

Limitations stated by the authors (Section V)
  • The model struggles to use wrist-camera information; finetuning was often better with only a third-person camera.
  • Language-conditioned and goal-conditioned performance differ greatly, and the authors attribute both to data: only 27% of the data has wrist cameras and only 56% has language.
  • The model learns by imitation of optimal demonstrations; sub-optimal or online data would need other objectives.
  • Only single- and dual-arm manipulators were tested; navigation and mobile manipulation are future work.

Part F

Critical paper review

Everything above reports what the paper says. This part is our independent reading: where its numbers and counts disagree, where its comparisons differ in more than the thing being tested, and where small sample sizes limit the conclusions. Octo is an open, reproducible and useful piece of work, and its design is clearly described; the findings concern the strength of the evidence. Each one links back to the concept it affects, and the calculations are reproducible in the examples.

Numbers that do not agree

  1. F1. “20 trials per domain”, but 10 for Bimanual Numerical inconsistency

    What the paper says. Table I’s caption says that in each domain the success rates are averaged over 20 trials.

    What we found. Appendix F gives 20 trials for Baking, Coffee, Insertion and Pick-Up (the Coke count is not given), but 10 for Berkeley Bimanual. The Bimanual values (20, 50, 80) are multiples of 10, which fits 10 trials. Pooling the trials with Bimanual at 10 trials gives 71.8% for Octo and 12.7% for VC-1, against 72.5% and 15.8% with 20; the printed averages are plain means of the six domains.

    Why it matters. The headline averages weight a 10-trial domain like a 20-trial one, and the caption misstates the trial count. Concept: Finetuning · example

  2. F2. Table I truncates its averages; the ablation tables round Numerical inconsistency

    What the paper says. Table I gives averages of 20, 15 and 72 over six domains; “Octo outperforms the next best baseline by 52%”.

    What we found. The six VC-1 values average 15.83 and the six Octo values 72.5; they are printed as 15 and 72, which is truncation. Tables II and VI round half up (82.5 as 83, 42.5 as 43, 17.5 as 18). With the same rounding, Table I would read 16 and 73, and the lead 53 points. The 52% is a difference in percentage points (72 minus 20); as a ratio it is 3.6 times.

    Why it matters. A one-point inconsistency, but “52%” can be read as a relative improvement of 52%, which would be a very different claim. Concept: Finetuning · example

  3. F3. Parameter counts leave out the 111-million-parameter language encoder Numerical inconsistency

    What the paper says. Octo-Small has 27 million parameters and Octo-Base 93 million; Octo (93M) is compared with RT-1-X (35M) and RT-2-X (55B).

    What we found. Table V and our estimate of the transformer blocks (84.9 million for Octo-Base) show that the stated counts are the trained network. The frozen T5-base text encoder, with 111 million parameters, is separate, so the model that runs for a language-conditioned task has about 204 million parameters, 5.8 times RT-1-X rather than 2.7 times.

    Why it matters. The size comparison with RT-1-X is fair for what is trained but understates what is needed at run time; a goal-image-only use avoids the encoder. Concept: Sizes and cost · example

How far the claims go

  1. F4. Many of the gaps are within chance Method caveat

    What the paper says. Octo-Small is best when using the ViT architecture, diffusion head and wide data mixture; Octo improves with size; Octo outperforms baselines in finetuning.

    What we found. Each result is one run: 10 trials per task for the zero-shot and scaling tests, 20 per finetuning domain, 40 per ablation row. With 40 trials, the 13-point gap between Octo-Small and the ResNet-50 variant (33/40 against 28/40) has an exact p-value of 0.293 and the 22.5-point gap to the 11-dataset mix 0.047. In finetuning, Coffee (15/20 against 9/20, p of 0.105) and Baking (10/20 against 5/20, 0.191) are within chance, while Insertion, Pick-Up and Coke are far beyond it. The scaling figure uses 10 trials of one task per robot.

    Why it matters. The headline conclusions (diffusion over regression and discretised heads, pretraining over scratch in total) are supported; the finer claims about encoders, data-mix size and model size are suggestions. Concept: The set-up · Ablations · example · example

  2. F5. The finetuning baselines differ in more than pretraining Method caveat

    What the paper says. Octo is a better initialisation than training from scratch or using pretrained visual representations such as VC-1.

    What we found. The from-scratch policy is a different architecture (ResNet encoder, FiLM language conditioning, a small transformer), used because the ViT overfitted from scratch, and VC-1 is combined with a mean-squared-error decoder, a loss that Octo’s own ablation puts at 35% against 83% for the diffusion head. There is no row for Octo’s architecture trained from scratch or for a VC-1 encoder with a diffusion head, so the contributions of pretraining, architecture and loss are not separated.

    Why it matters. The comparison shows that the Octo recipe beats commonly used alternatives; it overstates the gain attributable to the pretrained weights alone. Concept: The set-up · Finetuning

  3. F6. “Novel objects” and “in-distribution” use different kinds of task Method caveat

    What the paper says. Octo performs best on in-distribution tasks and generalises well to novel objects; goal images gave 25% higher success than language on the WidowX tasks.

    What we found. The two in-distribution tasks of Table VII are the language-conditioned tasks of Table VI and the two novel-object tasks are the goal-image tasks (Appendix F says the goal tasks contain objects not seen in Bridge). So the comparison changes both the novelty and the way the task is specified. In Octo-Small’s own ablation row the language tasks average 85% and the goal-image tasks 80%, so goal images are not 25% higher there; the 25% refers to a different zero-shot comparison whose values are not printed.

    Why it matters. Whether novel objects are handled “well” cannot be separated from the conditioning mode, and the goal-image advantage is not general. Concept: Zero-shot · Ablations · example

  4. F7. The zero-shot headline numbers are not printed Limitation

    What the paper says. Octo has a 29% higher success rate than RT-1-X on average and performs similarly to RT-2-X.

    What we found. The values appear only as bars in Figure 5. The RT-2-X WidowX numbers are copied from another paper, and the RT-1 robot evaluation was run by the authors of RT-2-X, so protocols may differ. “29% higher” is not stated as points or ratio, and “similar” has no definition. The tasks come from the pretraining datasets, and 10 trials per task give two tasks per robot.

    Why it matters. The ranking against RT-1-X is plausible, but the size of the gap and the claim of parity with a 55-billion-parameter model cannot be checked from the text. Concept: Zero-shot

  5. F8. A “wide” mixture dominated by three datasets Ambiguity

    What the paper says. The best models are those trained on the widest data mixtures; Octo is trained on the most diverse robot dataset to date.

    What we found. Fractal, Kuka and Bridge make up 51.0% of every batch, the top five 66.1%; the single-robot and RT-X comparisons confound diversity with volume, since the smaller mixes have fewer datasets and fewer samples. The data ablation is on one robot, the WidowX, whose own data is 17% of the mixture. The authors note that only 27% of the data has a wrist camera and 56% language.

    Why it matters. The result supports “more data from more robots helped on the WidowX”, not that breadth in general is the cause, and the weaknesses the authors report follow from the imbalance. Concept: The data · Ablations · example

  6. F9. “First” claims that the paper cannot check Ambiguity

    What the paper says. Octo is the first generalist robot policy that can be effectively finetuned to new observation and action spaces, and the first fully open-source generalist manipulation policy, including the pipeline, checkpoints and data.

    What we found. The paper offers no survey of all earlier models. It also uses the released RT-1-X checkpoint as the baseline, calling it the best openly available policy, so openness here means the full pipeline and data as well as weights. The finetuning claim rests on four setups with new inputs, action spaces or robots, about 100 demonstrations each.

    Why it matters. The openness is concrete and checkable; the “first” statements are the authors’ positioning at the time. Concept: Adapting to new inputs and outputs · Sizes and cost

What is not tested

  1. F10. Proprioception and wrist cameras: weak in pretraining, needed in finetuning Ambiguity

    What the paper says. Policies trained with proprioceptive inputs seemed generally worse; the model struggles to use wrist-camera information.

    What we found. Pretraining uses no joint readings, and wrist cameras exist in only 27% of the data, yet four of the six finetuning setups depend on them: joint readings for Pick-Up, force and torque for Insertion, wrist cameras for Coke and Bimanual. The paper does not test those tasks without the added inputs, so how much of the finetuning success comes from the new inputs, or whether a third-person camera alone would do as well, is not shown.

    Why it matters. The “new observation spaces” result is encouraging, but the paper’s own appendix suggests these inputs are used less well than the main text implies. Concept: Adapting to new inputs and outputs · What did not work

  2. F11. What the paper says about its limits, and what it leaves out Limitation

    What the paper says. The authors list weak wrist-camera use, a large gap between language and goal conditioning, imitation only, and manipulators only.

    What we found. These are accurate. The paper does not discuss that every result is a single run, that the zero-shot tasks come from the pretraining data, that language following is hardly tested (the two language tasks of the ablation, and finetuning tasks that are not described as language-conditioned), or that no compute-matched comparison with a vision-language policy such as RT-2-X is given. Evaluations were run by the model team and collaborators on their own robots.

    Why it matters. Octo is a solid open starting point for finetuning; the evidence supports “finetunes well to new setups” more than “generalises zero-shot to new tasks”. Concept: Zero-shot · Finetuning

Knowledge check

Try each question first, then reveal the explanation.

Question 1

In your own words: how does Octo take different robots and tasks with one network?

Explanation

Every input, whether a sentence, a goal image or a camera frame, becomes tokens in one sequence. A block-wise attention mask lets inputs be present or missing, readout tokens collect the answer without disturbing the rest, and a small diffusion head turns the readout into a chunk of actions. New inputs and outputs are attached with small new encoders and heads.

Question 2

Why does Octo use a diffusion head, and what does it cost?

Explanation

It can represent several good actions instead of their average, and is more precise than discretised actions. It costs 20 denoising steps per chunk, but they run inside a small head, after a single pass of the large transformer.

Question 3

What does the finetuning table show, and what does it not separate?

Explanation

Octo reaches 72% on average against 20% for a from-scratch policy and 15% for VC-1 with about 100 demonstrations. It does not separate pretraining from architecture and loss, because the baselines differ in both.

Question 4

Name one place where the paper’s numbers or counts disagree.

Explanation

For example: Table I’s caption says 20 trials per domain but Appendix F gives 10 for Bimanual; or Table I averages are truncated (15.83 as 15, 72.5 as 72) while Table II rounds half up; or “52%” is a difference of 52 points.

Glossary

Generalist robot policy (GRP)
A policy pretrained on many robots and tasks, meant to be used on several of them and finetuned to new ones.
Open X-Embodiment
A pooled collection of robot datasets from many labs; Octo trains on 25 of them.
Token
A small vector the transformer reads; text, image patches and readouts are all tokens.
Patch
A small square piece of an image (16 by 16 pixels here) turned into one token.
Block-wise attention
A mask that says which groups of tokens may look at which others.
Readout token
A learned token that collects a summary of the observations and is not read by other tokens.
Diffusion head
A small network that turns noise into an action chunk through several denoising steps.
DDPM
Denoising diffusion probabilistic model: the standard way of training and sampling diffusion models.
Action chunk
A short sequence of future actions predicted together.
Goal image
A picture of the desired end state, used instead of a sentence to say the task.
Hindsight relabelling
Using a later state of the same trajectory as the goal for an earlier example.
Zero-shot
Using the pretrained model without finetuning.
Finetuning
Continuing to train a pretrained model on a small dataset of the new setup.
Proprioception
The robot’s own readings of its joints, forces or torques.
Delta end-effector control
Actions given as small changes of the hand’s position and rotation.
Mean squared error (MSE)
A loss that rewards predicting the average of the demonstrations.
VC-1
A pretrained visual representation for robotics, used as a baseline.
RT-1-X and RT-2-X
Models of the Open X-Embodiment project: a 35-million-parameter transformer and a 55-billion-parameter vision-language model, both trained on Open X-Embodiment data.
Trial
One attempt at a task; success rates here are out of 10 to 40 trials.
Standard error
How much a measured rate would typically move by chance if the trials were repeated.

References

Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

  1. Octo Model Team (Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., et al.) (2024). Octo: An open-source generalist robot policy. RSS 2024 / arXiv. arXiv:2405.12213

    Cited for:

    • A transformer policy that turns language instructions or goal images and camera images into tokens (a frozen T5-base text encoder and a shallow convolutional patch stem), uses block-wise attention with readout tokens, and decodes chunks of actions with a small diffusion head of 20 denoising steps; new inputs and action heads can be added at finetuning without changing the pretrained transformer. Checked against the full text of the paper (arXiv v2 PDF), read sentence by sentence.
    • A large transformer-based policy trained on 800k trajectories from the Open X-Embodiment dataset, which can be instructed with language commands or goal images. Checked against the paper’s abstract.
    • Can be effectively finetuned to robot setups with new sensory inputs and action spaces within a few hours on standard consumer GPUs; tested across 9 robotic platforms. Checked against the paper’s abstract.

Go further

All paper guides · Back to the VLA track