M10-T6 — Foundation Models as Robot Policies: A Bridge to VLA

Learning objectives

After this lesson, you should be able to:

• Describe a vision-language-action (VLA) policy as a function from images, an instruction and state to actions, and name what is pretrained and what is trained on robot data.
• Relate pretrained policies to behaviour cloning (M3), RL fine-tuning (M6) and offline learning (M8), and say what each contributes and what fails in each.
• Compute action-quantisation resolution and a rough per-step compute budget for a token-based policy, and measure the effect of acting on a chunk of actions without feedback in the Scout toy.
• State what remains unproven about foundation-model policies and which evaluation, stress-test and monitoring practices from this track still apply.

The intuition

Everything in this track so far fits one frame: a policy maps observations to actions, and learning chooses its parameters from data (demonstrations, rewards, logs, a simulator). Scout's policy had three numbers. A foundation-model policy has the same job and a very different scale: the input is camera images plus a natural-language instruction plus perhaps robot state, the output is an action (or a short sequence of them), and the parameters number in the hundreds of millions to billions. The idea is that most of what the model needs (what objects look like, what words mean, how scenes are arranged) can be learned from large data that are not robot data, and the robot-specific part (which motor command does the thing) from a smaller amount of robot data.

The figure places this next to the rest of the track. The model in the middle is pretrained, then adapted to a robot. The ways of adapting are the ones you know. Imitation: train on demonstrations, supervised, exactly as in Module 3 but with a much larger network, richer inputs and many tasks. RL fine-tuning: improve the policy with reward, as in Module 6, which is the hardest to scale because rollouts on a real robot are expensive. Offline data: use logged experience of many robots and tasks, as in Module 8, with the same coverage and extrapolation questions. The model does not remove those questions; it changes the scale at which they appear.

Because I cannot run these models here, the rest of the lesson sticks to things I can state with confidence or can measure on the toy, and marks the rest.

The mathematics

The policy interface

In module notation, a language-conditioned policy is

I_t

Camera image(s) at time t; in M1 terms the observation o_t, now high-dimensional.

c

The instruction, a text string (the "goal" of M1, given in words).

s_t

Proprioceptive state (joint angles, gripper); may be absent.

H

Chunk length: how many future actions the model emits per call. H = 1 is the reactive policy of this track.

θ

Parameters, initialised from a model pretrained on images and text and often on robot data.

Relationship to earlier methods

Systems I can name (with the level of detail I am sure of)

• RT-1 (Brohan et al.): a transformer policy that takes camera images and a language instruction and outputs discretised actions, trained by imitation on a large set of real-robot demonstrations.
• RT-2 (Brohan et al.): starts from a pretrained vision-language model and trains it on robot data (together with web data) so that actions are produced as text-like tokens; the paper calls the result a vision-language-action model.
• Open X-Embodiment / RT-X: a collaboration pooling robot datasets from many labs and robot types, and models trained on them.
• Octo, OpenVLA and π0: open or publicly described generalist policies trained on large multi-robot datasets. I am sure they exist and are about this idea. Uncertain: their exact architectures (for instance which use action tokens and which use a diffusion or flow-matching action head), sizes and training data; I do not give details, and the references below are for the reader to check.

I deliberately give no success rates, no model sizes and no comparisons between these systems. I cannot verify numbers here and they change quickly.

A worked example

Two things can be worked out on Scout without any foundation model: what discretising actions costs, and what acting without feedback costs. Neither says anything about how well a real VLA works. Numbers come from Node (arithmetic, and RLCore episodes with the thetas of lesson M10-T4: tight = clear 0.3, kp 0; randomised = clear 0.9, kp 1; 5 seeds x 40 starts = 200 episodes per cell; scratch scripts).

By hand: action tokens and compute

Quantisation. Suppose each action component in [−1, 1] is discretised into 256 bins (a design used by token-based policies; I am sure that RT-1 discretised actions, the bin count is the example I use). One bin is 2/256 = 0.0078 wide, so rounding to the bin centre errs by at most 0.0039. On Scout (top speed 1 m/s, 0.25 s steps) that is at most 0.25 × 0.0039 ≈ 0.001 m, one millimetre, per step: negligible against the 0.25 m step. For a fine task (millimetre insertion) the same resolution is not negligible, and the number of bins is a design choice.

Compute. A dense transformer needs very roughly 2 floating-point operations per parameter per generated token (a standard rule of thumb; approximate). A hypothetical 7-billion-parameter model producing 7 action tokens per control step (one number per degree of freedom of a 7-degree-of-freedom command) needs 2 × 7 × 10⁹ = 1.4 × 10¹⁰ operations per token and 7 × 1.4 × 10¹⁰ ≈ 1.0 × 10¹¹ per step, ignoring the image encoder and the prompt. At Scout's 4 steps per second that is about 3.9 × 10¹¹ operations per second, to be set against the tens of operations per step of Scout's three-number policy. This is why such policies often emit a chunk of actions per call or run on a separate computer, and why latency enters the problem.

Measured: acting on a chunk without looking

If the model returns H actions per call, the robot executes them open-loop, without a new observation, until the next call. I imitate that in the toy by recomputing Scout's command only every H steps and holding it in between (same dynamics, same worlds; this is a toy proxy for chunking, not an implementation of any real system).

The cliff moves with the policy: the randomised policy tolerates two held steps in the nominal world and four on the floor, then crashes (at H = 8 on the floor, 99 of 200 episodes), while the tight policy breaks at H = 2 even in the nominal world (58 crashes of 200). The floor result being better than the nominal one for H = 4 is not noise I can explain away: the floor's lower gain (2.2 against 3) makes the robot slower, and a slower robot drifts less in four held steps. This is the same lesson as the delay sweep in lesson M10-T4, from a different cause, and it is a reason to test the chunk length the same way: a sweep with intervals, a requirement and a worst case. (These rates come from a different random stream than the tables of M10-T4, so the tight floor figure, 134 of 200, differs from the 145 of 200 there; with 200 episodes the 95% interval is about 0.60 to 0.73.)

Evaluating a policy with few trials

Real-robot evaluations of generalist policies often use tens of trials per task. 17 successes in 20 give a 95% Wilson interval of 0.64 to 0.95; 85 in 100, 0.77 to 0.91; 170 in 200, 0.79 to 0.89. A difference of five percentage points between two policies is invisible at n = 20, which is why the habits of lessons M1-T5 and M11-T2 matter more, not less, as models grow.

Engineering implications and limitations

What carries over

• Evaluation on held-out scenes and wording. The split question of lesson M2-T6 becomes: which objects, scenes, robots, lighting conditions and phrasings were in the pretraining or fine-tuning data? A benchmark whose tasks leaked into the training data measures memory.
• Stress tests. Lesson M10-T4 applies with new factors: lighting, camera pose, distractor objects, paraphrased instructions, plus latency and chunk length. Report curves, intervals, the worst case and what you did not test.
• Staged rollout and monitoring. Lesson M10-T5: shadow mode, speed limits, rollback rules, OOD monitors. An image-based monitor is a harder version of the Mahalanobis monitor (features come from a learned encoder and the same blind spots apply).
• A safety layer outside the model. The wrapper of lesson M9-T4 does not care how the command was produced. A language model that follows an instruction wrongly is a fault source like the glitchy expert, and probably a stranger one.
• Reward and specification. An instruction is a specification; the failure modes of Module 5 (the policy does what the words permit, not what you meant) are not ruled out, and uncertain in how they show up.

What is new

• Inputs are open-ended. A table-top instruction can be phrased in unboundedly many ways, so test coverage is a sampling problem over language and scenes.
• Failure analysis is harder. With three parameters you can read the policy; with billions you diagnose by experiment.
• Cost is outside the training loop. Inference compute, latency and the robot hours for demonstrations decide what is feasible (see the compute estimate above).
• Provenance matters. What the model saw in pretraining is usually only partly known to you; record the version, the data you fine-tuned on and the seeds, as in lesson M11-T3.

What remains unproven

These are my reading of open questions, not findings from this track. (1) How reliably pretrained policies work outside the conditions of the data they were adapted on, and whether scale or data diversity is the main lever. (2) Whether and how much RL fine-tuning improves them beyond more demonstrations. (3) How to certify behaviour of a policy driven by free-form language: a stress test samples a family, and language has no natural family. (4) How much transfers between very different robots (the premise of pooled datasets). (5) The cost of ownership: compute, data collection and the time to find failures. If you read a paper's claim on any of these, check the number of trials, the held-out conditions and the baselines of a fine-tuned smaller model, as lesson M11-T1 says.

What this lesson is not

It does not evaluate any foundation model, it makes no claim about their performance, and the toy numbers above are Scout's. The Scout study is a small simulation (module 10: a nominal world standing for simulation and a floor world standing for reality, no real robot data).

Common misconceptions

Engineering takeaways

1. A vision-language-action policy maps images, an instruction and state to actions (or chunks of actions); it is pretrained at scale and adapted to a robot by imitation, RL fine-tuning or offline learning, the methods of Modules 3, 6 and 8.
2. The questions of this track carry over: distribution shift, coverage, held-out evaluation with intervals, stress tests, staged rollout, monitoring and a safety layer outside the learned model.
3. In the Scout toy, 256-bin action tokens cost at most about a millimetre per step, a 7-billion-parameter model would need roughly 10^11 operations per control step (a rough estimate), and holding commands for H steps dropped the tight policy from 200 of 200 to 142 of 200 at H = 2 and the randomised one from 200 of 200 to 91 of 200 at H = 4 in the nominal world.
4. What is unproven includes outside-distribution reliability, the value of RL fine-tuning, certification under free-form language, cross-robot transfer and total cost; this lesson makes no performance claim about any model.
5. Module 10 is a toy study with no real robot data; the next module turns to doing such work honestly: baselines, seeds, statistics and reproducibility.

References and further reading

These are the primary sources for the systems named above; I describe them only at the level stated and give no numbers from them. Check venues and details before relying on them. The Scout numbers are my own toy prototype.

• A. Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale”, Robotics: Science and Systems (RSS), 2023.
• A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, Conference on Robot Learning (CoRL), 2023.
• Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models”, IEEE International Conference on Robotics and Automation (ICRA), 2024.
• Octo Model Team, “Octo: An Open-Source Generalist Robot Policy”, Robotics: Science and Systems (RSS), 2024.
• M. J. Kim et al., “OpenVLA: An Open-Source Vision-Language-Action Model”, arXiv, 2024.
• K. Black et al., “π0: A Vision-Language-Action Flow Model for General Robot Control”, arXiv, 2024.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Module 10 closes the loop from simulation to deployment. Module 11 turns to the practice of doing this work honestly: baselines and ablations, seeds and statistics, reproducibility, and writing it up.

Related content

Nothing below is required to finish this lesson.

• Before
  M10-T5: Safe Deployment and Monitoring

  Stage a rollout with limits, monitors and rollback, and detect when the policy leaves its training distribution.