Vision-Language-Action (VLA)

Theory

Short engineering lessons that build the mental model behind the labs.

1. M1.1 Linear algebra: vectors, matrices and matrix multiplication Lesson · about 12 minutes to read How data is represented as arrays and how layers transform it.
2. M1.2 Calculus: derivatives, gradients and the chain rule Lesson · about 14 minutes to read How a small change in a parameter changes the loss, and how that flows backwards through a network.
3. M1.3 Probability and basic statistics Lesson · about 14 minutes to read Distributions, expectation and variance, which underlie noise, uncertainty and loss functions.
4. M1.4 Optimization and gradient descent Lesson · about 14 minutes to read Minimising a loss by repeatedly stepping against its gradient.
5. M2.1 Python fundamentals Lesson · about 12 minutes to read The language features needed to read and write model and data code.
6. M2.2 NumPy and data manipulation Lesson · about 13 minutes to read Working with arrays, shapes and indexing.
7. M2.3 PyTorch tensors and tensor operations Lesson · about 13 minutes to read The tensor library most robot-learning code is written in.
8. M2.4 Neural networks and backpropagation Lesson · about 14 minutes to read How a network's weights are updated from the loss.
9. M2.5 Training loops, loss functions and GPU basics Lesson · about 14 minutes to read The cycle of forward pass, loss, backward pass and update, and running it on a GPU.
10. M3.1 Neural network fundamentals Lesson · about 13 minutes to read Layers, activations and what a network can represent.
11. M3.2 Embeddings and tokenization Lesson · about 13 minutes to read Turning text and other inputs into vectors a model can use.
12. M3.3 Attention and self-attention Lesson · about 14 minutes to read Letting each element of a sequence weigh the others.
13. M3.4 Transformer encoders and decoders Lesson · about 14 minutes to read How attention blocks are stacked into full models.
14. M3.5 Pretraining and fine-tuning Lesson · about 13 minutes to read Learning general features first, then adapting to a specific task.
15. M4.1 RGB images and image representations Lesson · about 12 minutes to read Pixels, channels and how an image becomes a tensor.
16. M4.2 Basic computer vision concepts Lesson · about 14 minutes to read Features, convolution and common vision tasks.
17. M4.3 Vision encoders and feature extraction Lesson · about 14 minutes to read Converting an image into compact features a model can use.
18. M4.4 LLM fundamentals Lesson · about 14 minutes to read What large language models are and how they generate text.
19. M4.5 CLIP and vision-language alignment Lesson · about 14 minutes to read Learning a shared space for images and text.
20. M4.6 Multimodal representations and visual grounding Lesson · about 15 minutes to read Combining modalities and linking words to what is visible.
21. M5.1 Robot state, observations and actions Lesson · about 13 minutes to read What a robot knows, what it senses and what it can command.
22. M5.2 Joint space and Cartesian space Lesson · about 13 minutes to read Describing a robot's pose by joint angles or by the position of its hand.
23. M5.3 Forward and inverse kinematics Lesson · about 15 minutes to read Mapping joint angles to hand pose, and the reverse.
24. M5.4 Coordinate frames and transformations Lesson · about 15 minutes to read Expressing positions in different frames and converting between them.
25. M5.5 End-effector poses and joint commands Lesson · about 14 minutes to read The two common ways to tell a robot where to move.
26. M5.6 Basic feedback control Lesson · about 14 minutes to read Using measured error to correct a robot's motion.
27. M6.1 Supervised learning Lesson · about 13 minutes to read Learning a mapping from inputs to targets using labelled examples.
28. M6.2 Behavior cloning and imitation learning Lesson · about 14 minutes to read Training a policy to copy demonstrated behaviour.
29. M6.3 Robot demonstrations and trajectory datasets Lesson · about 14 minutes to read How demonstrations are recorded and organised.
30. M6.4 Training and validation Lesson · about 14 minutes to read Holding out data to check that a model generalises.
31. M6.5 Policy, observation, state and action Lesson · about 13 minutes to read The vocabulary used to describe a learned controller.
32. M6.6 Introduction to reinforcement learning Lesson · about 15 minutes to read Learning from reward by trial and error.
33. 1.1 What is Physical AI? Lesson · about 15 minutes to read AI that perceives, decides and acts in the physical world, and why that is harder than software-only AI.
34. 1.2 What is a Vision-Language-Action model? Lesson · about 15 minutes to read A model that maps camera images and a language instruction directly to robot actions.
35. 1.3 LLM planner vs. VLM vs. VLA Lesson · about 14 minutes to read What each model type takes in, what it outputs, and where each fits in a robot system.
36. 1.4 VLA architecture and the perception-to-action pipeline Lesson · about 16 minutes to read The main components, from observation encoding through to the action output.
37. 1.5 Open-loop vs. closed-loop robot execution Lesson · about 13 minutes to read Executing a plan blindly versus re-observing and correcting while the robot moves.
38. 2.1 Transformer architecture and attention Lesson · about 14 minutes to read Tokens, self-attention and the transformer block that most VLAs are built on.
39. 2.2 Vision Transformer (ViT) Lesson · about 12 minutes to read Treating an image as a sequence of patches so a transformer can process it.
40. 2.3 CLIP and vision-language alignment Lesson · about 14 minutes to read Learning a shared space where images and text can be compared.
41. 2.4 Vision encoders, language models, embeddings and multimodal fusion Lesson · about 14 minutes to read How separate modalities become one sequence a model can reason over.
42. 2.5 How visual and language representations condition robot actions Lesson · about 13 minutes to read How an instruction and an image decide what the robot does next.
43. 3.1 Robot observations, actions and state representations Lesson · about 13 minutes to read What a policy sees and what it outputs, and how both are represented.
44. 3.2 Demonstrations, trajectories and robot datasets Lesson · about 12 minutes to read How robot experience is recorded, stored and organised for learning.
45. 3.3 Behavior cloning and imitation learning Lesson · about 15 minutes to read Training a policy to reproduce what a demonstrator did.
46. 3.4 Training objectives, loss functions and evaluation Lesson · about 13 minutes to read What the model is optimised for, and how to tell whether it actually works.
47. 3.5 Distribution shift and generalization Lesson · about 14 minutes to read Why a policy that works on its training data can fail in a slightly different setting.
48. 4.1 Joint-space and end-effector action representations Lesson · about 12 minutes to read Commanding joint angles versus commanding the gripper pose.
49. 4.2 Discrete action tokens vs. continuous actions Lesson · about 13 minutes to read Treating actions as words in a vocabulary versus predicting real-valued vectors.
50. 4.3 Action normalization and decoding Lesson · about 11 minutes to read Scaling actions for training and turning model outputs back into robot commands.
51. 4.4 Temporal action prediction and action chunking Lesson · about 11 minutes to read Predicting a short sequence of future actions at once instead of a single step.
52. 4.5 Action Chunking with Transformers (ACT) Lesson · about 12 minutes to read A transformer policy that predicts chunks of actions from demonstrations.
53. 4.6 Diffusion Policy Lesson · about 14 minutes to read Generating actions by gradually denoising random noise, conditioned on observations.
54. 4.7 Flow matching for action generation Lesson · about 14 minutes to read A related generative approach that learns a smooth path from noise to actions.
55. 4.8 FAST action tokenization Lesson · about 14 minutes to read Compressing action sequences into compact tokens a language-model-style policy can predict.
56. 5.1 RT-1 Lesson · about 12 minutes to read A transformer policy trained on a large set of real-robot demonstrations.
57. 5.2 RT-2 Lesson · about 13 minutes to read Reusing a pretrained vision-language model to output robot actions.
58. 5.3 Open X-Embodiment Lesson · about 11 minutes to read A shared dataset and collaboration pooling demonstrations from many robots.
59. 5.4 Vision-language pretraining and robot demonstration data Lesson · about 13 minutes to read How web-scale pretraining and robot data combine in training.
60. 5.5 Generalization across tasks and objects Lesson · about 13 minutes to read What generalization means for robot policies and how it is demonstrated.
61. 6.1 OpenVLA Lesson · about 14 minutes to read An open vision-language-action model that predicts discrete action tokens.
62. 6.2 Octo Lesson · about 12 minutes to read An open generalist robot policy built to be fine-tuned to new robots and tasks.
63. 6.3 π0 and π0.5 Lesson · about 13 minutes to read VLA models from Physical Intelligence that generate continuous actions with flow matching.
64. 6.4 SmolVLA Lesson · about 11 minutes to read A compact VLA designed to be accessible with modest compute.
65. 6.5 RDT-1B Lesson · about 11 minutes to read A diffusion-based foundation model for bimanual manipulation.
66. 6.6 Comparing architectures, action representations, training methods and compute Lesson · about 11 minutes to read A side-by-side comparison of the models above across the same dimensions.
67. 7.1 Understanding a pretrained VLA's inference pipeline Lesson · about 13 minutes to read What happens between receiving an observation and returning an action.
68. 7.2 Dataset preparation and preprocessing Lesson · about 13 minutes to read Formatting, cleaning and normalising demonstrations for training.
69. 7.3 Running pretrained policies Lesson · about 12 minutes to read Loading a released model and executing it on observations.
70. 7.4 Fine-tuning on robot demonstrations Lesson · about 14 minutes to read Adapting a pretrained model to a new robot or task with your own data.
71. 7.5 Training a small policy from scratch Lesson · about 15 minutes to read Building and training a minimal policy end to end to see every step.
72. 7.6 Model evaluation and failure analysis Lesson · about 12 minutes to read Measuring results and sorting failures into causes you can act on.
73. 7.7 Simulation-based manipulation experiments Lesson · about 14 minutes to read Running controlled manipulation experiments in simulation.
74. 8.1 LIBERO and other suitable simulation benchmarks Lesson · about 12 minutes to read Standard simulated task suites for comparing policies.
75. 8.2 Task success rate and repeatability Lesson · about 11 minutes to read Measuring success over enough trials to trust the number.
76. 8.3 Generalization to new objects, positions and instructions Lesson · about 12 minutes to read Testing how far a policy holds up when the setup changes.
77. 8.4 Perception, grounding and action failures Lesson · about 12 minutes to read Telling apart a model that misread the scene, misunderstood the instruction or moved wrongly.
78. 8.5 Recovery and robustness Lesson · about 13 minutes to read Whether a policy can correct mistakes and cope with disturbances.
79. 8.6 Inference latency and action frequency Lesson · about 12 minutes to read How model speed limits how often a robot can be commanded.
80. 9.1 Policy inference as a robotics component Lesson · about 13 minutes to read Running the model as a service with defined inputs, outputs and timing.
81. 9.2 Action adapters and robot-specific interfaces Lesson · about 12 minutes to read Translating a model's outputs into a particular robot's commands.
82. 9.3 Coordinate frames and camera calibration Lesson · about 13 minutes to read Keeping image, robot and world coordinates consistent.
83. 9.4 Joint limits and collision checking Lesson · about 14 minutes to read Rejecting or adjusting actions that would be unsafe or impossible.
84. 9.5 ROS 2 integration Lesson · about 13 minutes to read Connecting the policy to a robot through ROS 2 nodes and messages.
85. 9.6 Policy execution and low-level controller interfaces Lesson · about 13 minutes to read Handing policy actions to the controllers that move the joints.
86. 9.7 Safety monitors and fallback behavior Lesson · about 14 minutes to read Detecting problems at run time and stopping or falling back safely.
87. 10.1 NVIDIA GR00T N1 and N1.5 Lesson · about 12 minutes to read NVIDIA's foundation models for humanoid robots.
88. 10.2 Gemini Robotics Lesson · about 12 minutes to read Google DeepMind's robotics models built on Gemini.
89. 10.3 Figure Helix Lesson · about 11 minutes to read Figure's VLA system for humanoid robots.
90. 10.4 VLA policies for humanoid manipulation Lesson · about 12 minutes to read What changes when a VLA drives arms and hands on a humanoid body.
91. 10.5 Learned policies and whole-body control Lesson · about 13 minutes to read How learned policies combine with controllers that coordinate the whole body.
92. 10.6 Balance, contact constraints and embodiment-specific actions Lesson · about 14 minutes to read Keeping a humanoid upright and respecting contacts while it acts.
93. 10.7 Integrating a VLA with a Unitree G1 humanoid Lesson · about 12 minutes to read An integration case study on the Unitree G1.
94. 11.1 Reinforcement learning for robot policies Lesson · about 14 minutes to read Improving policies through trial and reward, alone or on top of imitation.
95. 11.2 VLA fine-tuning and data efficiency Lesson · about 14 minutes to read Getting good behaviour from less robot data.
96. 11.3 Cross-embodiment transfer Lesson · about 13 minutes to read Reusing one policy across robots with different bodies.
97. 11.4 World models and predictive action planning Lesson · about 14 minutes to read Predicting what will happen so a robot can plan before it acts.
98. 11.5 Hierarchical planning and task decomposition Lesson · about 14 minutes to read Splitting a long task into steps a policy can execute.
99. 11.6 Safety and reliable autonomous execution Lesson · about 13 minutes to read What it takes to trust a learned policy to run on its own.
100. 11.7 Current open research problems Lesson · about 13 minutes to read The questions the field has not yet answered.