FixTheRobot Home / VLA / Roadmap

Learning roadmap · curriculum in development

Vision-Language-Action (VLA) Learning Roadmap

A structured path from the first idea of Physical AI to modern vision-language-action models, simulation, deployment and humanoid robots. You will learn how a VLA turns camera images and a language instruction into robot actions, how those models are trained and evaluated, and how to connect a policy to a real robot safely.

Outline only. This is a curriculum outline. The lessons are being written one at a time, and none of them is published yet. Each topic below will link to its lesson once it exists.

  1. FoundationsPhases 1–3
  2. Models and policiesPhases 4–6
  3. Build and measurePhases 7–8
  4. Real robotsPhases 9–10
  5. FrontierPhase 11

Prerequisites

What is helpful to know before Phase 1. You do not need to finish all of it first: skip what you already know, and pick up the rest as you go.

  1. AI foundations

    Modules 1–3: the general machine-learning toolkit, with no robotics needed.

    • Module 1 EssentialMathematics for Deep Learning The maths behind how neural networks compute and learn. 4 topics · 4 lessons published
      1. M1.1

        Linear algebra: vectors, matrices and matrix multiplication

        How data is represented as arrays and how layers transform it.

        Open lesson · 12 min
      2. M1.2

        Calculus: derivatives, gradients and the chain rule

        How a small change in a parameter changes the loss, and how that flows backwards through a network.

        Open lesson · 14 min
      3. M1.3

        Probability and basic statistics

        Distributions, expectation and variance, which underlie noise, uncertainty and loss functions.

        Open lesson · 14 min
      4. M1.4

        Optimization and gradient descent

        Minimising a loss by repeatedly stepping against its gradient.

        Open lesson · 14 min
    • Module 2 EssentialPython and Deep Learning The programming tools and training workflow used to build and train models. 5 topics · 5 lessons published
      1. M2.1

        Python fundamentals

        The language features needed to read and write model and data code.

        Open lesson · 12 min
      2. M2.2

        NumPy and data manipulation

        Working with arrays, shapes and indexing.

        Open lesson · 13 min
      3. M2.3

        PyTorch tensors and tensor operations

        The tensor library most robot-learning code is written in.

        Open lesson · 13 min
      4. M2.4

        Neural networks and backpropagation

        How a network's weights are updated from the loss.

        Open lesson · 14 min
      5. M2.5

        Training loops, loss functions and GPU basics

        The cycle of forward pass, loss, backward pass and update, and running it on a GPU.

        Open lesson · 14 min
    • Module 3 EssentialDeep Learning and Transformers The model family that almost every modern VLA is built on. 5 topics · 5 lessons published
      1. M3.1

        Neural network fundamentals

        Layers, activations and what a network can represent.

        Open lesson · 13 min
      2. M3.2

        Embeddings and tokenization

        Turning text and other inputs into vectors a model can use.

        Open lesson · 13 min
      3. M3.3

        Attention and self-attention

        Letting each element of a sequence weigh the others.

        Open lesson · 14 min
      4. M3.4

        Transformer encoders and decoders

        How attention blocks are stacked into full models.

        Open lesson · 14 min
      5. M3.5

        Pretraining and fine-tuning

        Learning general features first, then adapting to a specific task.

        Open lesson · 13 min

  2. Multimodal foundations

    Module 4: how vision and language models fit together.

    • Module 4 RecommendedComputer Vision and Language Models How machines represent images and language, and how the two are connected. 6 topics · 6 lessons published
      1. M4.1

        RGB images and image representations

        Pixels, channels and how an image becomes a tensor.

        Open lesson · 12 min
      2. M4.2

        Basic computer vision concepts

        Features, convolution and common vision tasks.

        Open lesson · 14 min
      3. M4.3

        Vision encoders and feature extraction

        Converting an image into compact features a model can use.

        Open lesson · 14 min
      4. M4.4

        LLM fundamentals

        What large language models are and how they generate text.

        Open lesson · 14 min
      5. M4.5

        CLIP and vision-language alignment

        Learning a shared space for images and text.

        Open lesson · 14 min
      6. M4.6

        Multimodal representations and visual grounding

        Combining modalities and linking words to what is visible.

        Open lesson · 15 min

  3. Robotics foundations

    Module 5: the robot side of the problem.

    • Module 5 EssentialRobotics Fundamentals The robot concepts a policy must connect to: what it senses, how it moves and how it is controlled. 6 topics · 6 lessons published
      1. M5.1

        Robot state, observations and actions

        What a robot knows, what it senses and what it can command.

        Open lesson · 13 min
      2. M5.2

        Joint space and Cartesian space

        Describing a robot's pose by joint angles or by the position of its hand.

        Open lesson · 13 min
      3. M5.3

        Forward and inverse kinematics

        Mapping joint angles to hand pose, and the reverse.

        Open lesson · 15 min
      4. M5.4

        Coordinate frames and transformations

        Expressing positions in different frames and converting between them.

        Open lesson · 15 min
      5. M5.5

        End-effector poses and joint commands

        The two common ways to tell a robot where to move.

        Open lesson · 14 min
      6. M5.6

        Basic feedback control

        Using measured error to correct a robot's motion.

        Open lesson · 14 min

  4. Robot learning foundations

    Module 6: how policies are learned from robot data.

    • Module 6 Learn along the wayRobot Learning Fundamentals The learning setup used to train robot policies from data. 6 topics · 6 lessons published
      1. M6.1

        Supervised learning

        Learning a mapping from inputs to targets using labelled examples.

        Open lesson · 13 min
      2. M6.2

        Behavior cloning and imitation learning

        Training a policy to copy demonstrated behaviour.

        Open lesson · 14 min
      3. M6.3

        Robot demonstrations and trajectory datasets

        How demonstrations are recorded and organised.

        Open lesson · 14 min
      4. M6.4

        Training and validation

        Holding out data to check that a model generalises.

        Open lesson · 14 min
      5. M6.5

        Policy, observation, state and action

        The vocabulary used to describe a learned controller.

        Open lesson · 13 min
      6. M6.6

        Introduction to reinforcement learning

        Learning from reward by trial and error.

        Open lesson · 15 min

The phases, in order

Each phase builds on the ones before it. Open a phase to see its topics.

  1. Phase 1Introduction to VLA What a VLA is, how it differs from a language planner or a vision-language model, and the basic shape of the perception-to-action pipeline.
    • Physical AI
    • VLA
    • LLM vs VLM vs VLA
    • Pipeline
    • Closed loop
    5 topics · 5 lessons published
    1. 1.1

      What is Physical AI?

      AI that perceives, decides and acts in the physical world, and why that is harder than software-only AI.

      Open lesson · 15 min
    2. 1.2

      What is a Vision-Language-Action model?

      A model that maps camera images and a language instruction directly to robot actions.

      Open lesson · 15 min
    3. 1.3

      LLM planner vs. VLM vs. VLA

      What each model type takes in, what it outputs, and where each fits in a robot system.

      Open lesson · 14 min
    4. 1.4

      VLA architecture and the perception-to-action pipeline

      The main components, from observation encoding through to the action output.

      Open lesson · 16 min
    5. 1.5

      Open-loop vs. closed-loop robot execution

      Executing a plan blindly versus re-observing and correcting while the robot moves.

      Open lesson · 13 min
  2. Phase 2Vision-Language Foundations The building blocks inside every VLA: transformers, vision encoders, language models and how they are fused.
    • Attention
    • ViT
    • CLIP
    • Fusion
    5 topics · 5 lessons published
    1. 2.1

      Transformer architecture and attention

      Tokens, self-attention and the transformer block that most VLAs are built on.

      Open lesson · 14 min
    2. 2.2

      Vision Transformer (ViT)

      Treating an image as a sequence of patches so a transformer can process it.

      Open lesson · 12 min
    3. 2.3

      CLIP and vision-language alignment

      Learning a shared space where images and text can be compared.

      Open lesson · 14 min
    4. 2.4

      Vision encoders, language models, embeddings and multimodal fusion

      How separate modalities become one sequence a model can reason over.

      Open lesson · 14 min
    5. 2.5

      How visual and language representations condition robot actions

      How an instruction and an image decide what the robot does next.

      Open lesson · 13 min
  3. Phase 3Robot Learning Foundations How robots learn from data: what is recorded, how a policy is trained from demonstrations, and why it can fail on new situations.
    • State and actions
    • Demonstrations
    • Behavior cloning
    • Distribution shift
    5 topics · 5 lessons published
    1. 3.1

      Robot observations, actions and state representations

      What a policy sees and what it outputs, and how both are represented.

      Open lesson · 13 min
    2. 3.2

      Demonstrations, trajectories and robot datasets

      How robot experience is recorded, stored and organised for learning.

      Open lesson · 12 min
    3. 3.3

      Behavior cloning and imitation learning

      Training a policy to reproduce what a demonstrator did.

      Open lesson · 15 min
    4. 3.4

      Training objectives, loss functions and evaluation

      What the model is optimised for, and how to tell whether it actually works.

      Open lesson · 13 min
    5. 3.5

      Distribution shift and generalization

      Why a policy that works on its training data can fail in a slightly different setting.

      Open lesson · 14 min
  4. Phase 4Action Representation and Policy Learning How actions are represented and generated, from tokens to chunks to diffusion and flow matching.
    • Action tokens
    • Chunking
    • ACT
    • Diffusion Policy
    • Flow matching
    • FAST
    8 topics · 8 lessons published
    1. 4.1

      Joint-space and end-effector action representations

      Commanding joint angles versus commanding the gripper pose.

      Open lesson · 12 min
    2. 4.2

      Discrete action tokens vs. continuous actions

      Treating actions as words in a vocabulary versus predicting real-valued vectors.

      Open lesson · 13 min
    3. 4.3

      Action normalization and decoding

      Scaling actions for training and turning model outputs back into robot commands.

      Open lesson · 11 min
    4. 4.4

      Temporal action prediction and action chunking

      Predicting a short sequence of future actions at once instead of a single step.

      Open lesson · 11 min
    5. 4.5

      Action Chunking with Transformers (ACT)

      A transformer policy that predicts chunks of actions from demonstrations.

      Open lesson · 12 min
    6. 4.6

      Diffusion Policy

      Generating actions by gradually denoising random noise, conditioned on observations.

      Open lesson · 14 min
    7. 4.7

      Flow matching for action generation

      A related generative approach that learns a smooth path from noise to actions.

      Open lesson · 14 min
    8. 4.8

      FAST action tokenization

      Compressing action sequences into compact tokens a language-model-style policy can predict.

      Open lesson · 14 min
  5. Phase 5Foundational VLA Research The research that established VLAs: large robot datasets, and transferring web-scale vision-language knowledge to control.
    • RT-1
    • RT-2
    • Open X-Embodiment
    5 topics · 5 lessons published
    1. 5.1

      RT-1

      A transformer policy trained on a large set of real-robot demonstrations.

      Open lesson · 12 min
    2. 5.2

      RT-2

      Reusing a pretrained vision-language model to output robot actions.

      Open lesson · 13 min
    3. 5.3

      Open X-Embodiment

      A shared dataset and collaboration pooling demonstrations from many robots.

      Open lesson · 11 min
    4. 5.4

      Vision-language pretraining and robot demonstration data

      How web-scale pretraining and robot data combine in training.

      Open lesson · 13 min
    5. 5.5

      Generalization across tasks and objects

      What generalization means for robot policies and how it is demonstrated.

      Open lesson · 13 min
  6. Phase 6Modern VLA Models The current generation of VLA models, compared by architecture, action representation, training method and compute needs.
    • OpenVLA
    • Octo
    • π0
    • SmolVLA
    • RDT-1B
    6 topics · 6 lessons published
    1. 6.1

      OpenVLA

      An open vision-language-action model that predicts discrete action tokens.

      Open lesson · 14 min
    2. 6.2

      Octo

      An open generalist robot policy built to be fine-tuned to new robots and tasks.

      Open lesson · 12 min
    3. 6.3

      π0 and π0.5

      VLA models from Physical Intelligence that generate continuous actions with flow matching.

      Open lesson · 13 min
    4. 6.4

      SmolVLA

      A compact VLA designed to be accessible with modest compute.

      Open lesson · 11 min
    5. 6.5

      RDT-1B

      A diffusion-based foundation model for bimanual manipulation.

      Open lesson · 11 min
    6. 6.6

      Comparing architectures, action representations, training methods and compute

      A side-by-side comparison of the models above across the same dimensions.

      Open lesson · 11 min
  7. Phase 7VLA Implementation and Experimentation Hands-on work: run a pretrained policy, prepare data, fine-tune, train a small policy and analyse where it fails.
    • Inference
    • Datasets
    • Fine-tuning
    • Failure analysis
    • Simulation
    7 topics · 7 lessons published
    1. 7.1

      Understanding a pretrained VLA's inference pipeline

      What happens between receiving an observation and returning an action.

      Open lesson · 13 min
    2. 7.2

      Dataset preparation and preprocessing

      Formatting, cleaning and normalising demonstrations for training.

      Open lesson · 13 min
    3. 7.3

      Running pretrained policies

      Loading a released model and executing it on observations.

      Open lesson · 12 min
    4. 7.4

      Fine-tuning on robot demonstrations

      Adapting a pretrained model to a new robot or task with your own data.

      Open lesson · 14 min
    5. 7.5

      Training a small policy from scratch

      Building and training a minimal policy end to end to see every step.

      Open lesson · 15 min
    6. 7.6

      Model evaluation and failure analysis

      Measuring results and sorting failures into causes you can act on.

      Open lesson · 12 min
    7. 7.7

      Simulation-based manipulation experiments

      Running controlled manipulation experiments in simulation.

      Open lesson · 14 min
  8. Phase 8VLA Evaluation and Generalization How to measure a VLA fairly: benchmarks, success rates, generalization tests, failure types, robustness and speed.
    • LIBERO
    • Success rate
    • Generalization
    • Latency
    6 topics · 6 lessons published
    1. 8.1

      LIBERO and other suitable simulation benchmarks

      Standard simulated task suites for comparing policies.

      Open lesson · 12 min
    2. 8.2

      Task success rate and repeatability

      Measuring success over enough trials to trust the number.

      Open lesson · 11 min
    3. 8.3

      Generalization to new objects, positions and instructions

      Testing how far a policy holds up when the setup changes.

      Open lesson · 12 min
    4. 8.4

      Perception, grounding and action failures

      Telling apart a model that misread the scene, misunderstood the instruction or moved wrongly.

      Open lesson · 12 min
    5. 8.5

      Recovery and robustness

      Whether a policy can correct mistakes and cope with disturbances.

      Open lesson · 13 min
    6. 8.6

      Inference latency and action frequency

      How model speed limits how often a robot can be commanded.

      Open lesson · 12 min
  9. Phase 9VLA Deployment and Robot Integration Turning a policy into a safe robot component: interfaces, frames, calibration, limits, ROS 2 and fallbacks.
    • ROS 2
    • Frames
    • Limits
    • Safety monitors
    7 topics · 7 lessons published
    1. 9.1

      Policy inference as a robotics component

      Running the model as a service with defined inputs, outputs and timing.

      Open lesson · 13 min
    2. 9.2

      Action adapters and robot-specific interfaces

      Translating a model's outputs into a particular robot's commands.

      Open lesson · 12 min
    3. 9.3

      Coordinate frames and camera calibration

      Keeping image, robot and world coordinates consistent.

      Open lesson · 13 min
    4. 9.4

      Joint limits and collision checking

      Rejecting or adjusting actions that would be unsafe or impossible.

      Open lesson · 14 min
    5. 9.5

      ROS 2 integration

      Connecting the policy to a robot through ROS 2 nodes and messages.

      Open lesson · 13 min
    6. 9.6

      Policy execution and low-level controller interfaces

      Handing policy actions to the controllers that move the joints.

      Open lesson · 13 min
    7. 9.7

      Safety monitors and fallback behavior

      Detecting problems at run time and stopping or falling back safely.

      Open lesson · 14 min
  10. Phase 10Humanoid VLA Systems VLA policies on humanoid robots, where manipulation must work together with balance and whole-body control.
    • GR00T
    • Gemini Robotics
    • Helix
    • Whole-body control
    • Unitree G1
    7 topics · 7 lessons published
    1. 10.1

      NVIDIA GR00T N1 and N1.5

      NVIDIA's foundation models for humanoid robots.

      Open lesson · 12 min
    2. 10.2

      Gemini Robotics

      Google DeepMind's robotics models built on Gemini.

      Open lesson · 12 min
    3. 10.3

      Figure Helix

      Figure's VLA system for humanoid robots.

      Open lesson · 11 min
    4. 10.4

      VLA policies for humanoid manipulation

      What changes when a VLA drives arms and hands on a humanoid body.

      Open lesson · 12 min
    5. 10.5

      Learned policies and whole-body control

      How learned policies combine with controllers that coordinate the whole body.

      Open lesson · 13 min
    6. 10.6

      Balance, contact constraints and embodiment-specific actions

      Keeping a humanoid upright and respecting contacts while it acts.

      Open lesson · 14 min
    7. 10.7

      Integrating a VLA with a Unitree G1 humanoid

      An integration case study on the Unitree G1.

      Open lesson · 12 min
  11. Phase 11Advanced Research Directions Where the field is heading and what is still unsolved.
    • RL
    • Cross-embodiment
    • World models
    • Open problems
    7 topics · 7 lessons published
    1. 11.1

      Reinforcement learning for robot policies

      Improving policies through trial and reward, alone or on top of imitation.

      Open lesson · 14 min
    2. 11.2

      VLA fine-tuning and data efficiency

      Getting good behaviour from less robot data.

      Open lesson · 14 min
    3. 11.3

      Cross-embodiment transfer

      Reusing one policy across robots with different bodies.

      Open lesson · 13 min
    4. 11.4

      World models and predictive action planning

      Predicting what will happen so a robot can plan before it acts.

      Open lesson · 14 min
    5. 11.5

      Hierarchical planning and task decomposition

      Splitting a long task into steps a policy can execute.

      Open lesson · 14 min
    6. 11.6

      Safety and reliable autonomous execution

      What it takes to trust a learned policy to run on its own.

      Open lesson · 13 min
    7. 11.7

      Current open research problems

      The questions the field has not yet answered.

      Open lesson · 13 min

← Back to the VLA track