FixTheRobot Explore Robotics / Robot Learning

Learning track

Robot Learning

How robots learn behaviours from data, demonstrations and trial and error, and how to tell why a learned policy succeeds or fails. Theory lessons for the ideas, labs where you train, evaluate, diagnose and fix.

65 of 65 theory lessons and 37 of 37 labs are open. Items marked planned are not available yet. Track status: in development. Start with M1-T1: What Is Robot Learning?.

One robot, one world, many learners

Every lab is an experiment on Scout, a delivery robot at a fictional hospital, in one small simulated world. The same task is attacked with imitation, Q-learning, policy gradients and a learned model, so you can compare methods on equal terms. The running question is always why did the policy fail, and how do you know?

Each module ends with a diagnose-and-fix lab. The browser simulations are deliberately small and say what they simplify; full-scale PyTorch and simulator experiments are offered as projects for your own machine and are labelled as not run by this site's tests.

Module 1 · Complete

Foundations: Policies, Data and Evaluation

What it means for a robot to learn: policies, observations, actions, rewards and datasets, the difference between training and deployment, and how to tell whether a learned policy is actually good.

Prerequisites: none.

Completion criteria
  • A learner can define a policy, build a dataset and a return from a rollout, and evaluate a policy with a confidence interval on held-out scenarios.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M1-T1What Is Robot Learning?Theory
  2. M1-T2Observations, Actions and DatasetsTheory
  3. M1-T3Rewards, Returns and ObjectivesTheory
  4. M1-T4Training Versus InferenceTheory
  5. M1-T5Evaluating a PolicyTheory
  6. M1-T6Why Learned Policies FailTheory

Labs (4)

  1. M1-L1Record a RolloutBrowser lab
  2. M1-L2Learn to Reach the GoalBrowser lab
  3. M1-L3Score a Policy FairlyBrowser lab
  4. M1-L4Read a Failed Training RunBrowser tool

Module 2 · Complete

Supervised Learning for Robots

Regression and classification from robot data: features, loss functions, gradient descent, small neural networks, generalization and the leaks that make a bad model look good.

Prerequisites: M1.

Completion criteria
  • A learner can fit and evaluate a regression and a small neural network on robot data, and detect overfitting and data leakage.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M2-T1Regression and Classification for RobotsTheory
  2. M2-T2Features and NormalizationTheory
  3. M2-T3Loss Functions and Gradient DescentTheory
  4. M2-T4Neural Networks from ScratchTheory
  5. M2-T5Generalization and OverfittingTheory
  6. M2-T6Splits, Leakage and Robot DataTheory

Labs (3)

  1. M2-L1Fit Steering with Gradient DescentBrowser lab
  2. M2-L2Train a Tiny Neural NetworkBrowser lab
  3. M2-L3Catch the Overfit ModelBrowser lab

Module 3 · Complete

Imitation Learning

Learning a policy from expert demonstrations: behaviour cloning, why it drifts, interactive correction with DAgger, and the quality of demonstrations.

Prerequisites: M1, M2.

Completion criteria
  • A learner can clone a demonstrated behaviour, diagnose compounding errors from distribution shift, and improve the policy with corrections and cleaner data.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M3-T1Learning from DemonstrationsTheory
  2. M3-T2Behavior CloningTheory
  3. M3-T3Distribution Shift and Compounding ErrorsTheory
  4. M3-T4DAgger and Interactive ImitationTheory
  5. M3-T5Collecting Good DemonstrationsTheory
  6. M3-T6Multimodal Actions and Beyond Mean Squared ErrorTheory

Labs (4)

  1. M3-L1Clone the ExpertBrowser lab
  2. M3-L2Watch the Clone DriftBrowser lab
  3. M3-L3Add DAgger CorrectionsBrowser lab
  4. M3-L4Clean the DemonstrationsBrowser lab

Module 4 · Complete

Reinforcement Learning Foundations

Learning by trial and error: Markov decision processes, returns, value functions and the Bellman equations, dynamic programming, temporal-difference learning, Q-learning and exploration.

Prerequisites: M1.

Completion criteria
  • A learner can write an MDP, solve it with value iteration, implement tabular Q-learning with exploration, and diagnose a Q-learning agent that does not learn.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (7)

  1. M4-T1Markov Decision ProcessesTheory
  2. M4-T2Returns and DiscountingTheory
  3. M4-T3Value Functions and the Bellman EquationsTheory
  4. M4-T4Dynamic Programming and Value IterationTheory
  5. M4-T5Monte Carlo and Temporal-Difference LearningTheory
  6. M4-T6Q-Learning and SARSATheory
  7. M4-T7Exploration Versus ExploitationTheory

Labs (4)

  1. M4-L1Solve an MDP by Value IterationBrowser lab
  2. M4-L2Implement Q-LearningBrowser lab
  3. M4-L3Tune ExplorationBrowser tool
  4. M4-L4Debug a Q-Learning AgentBrowser lab

Module 5 · Complete

Reward Design

What a reward really specifies: shaping, sparse versus dense rewards, potential-based shaping, reward hacking, constraints and multi-objective trade-offs.

Prerequisites: M1, M4.

Completion criteria
  • A learner can shape a reward without changing the optimal behaviour, find and fix a reward that is being exploited, and balance reward terms against constraints.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (5)

  1. M5-T1What a Reward Really SaysTheory
  2. M5-T2Reward Shaping and Potential FunctionsTheory
  3. M5-T3Sparse Versus Dense RewardsTheory
  4. M5-T4Reward Hacking and Specification GamingTheory
  5. M5-T5Constraints, Safety and Multi-Objective RewardsTheory

Labs (3)

  1. M5-L1Shape a Reward Without Changing the TaskBrowser lab
  2. M5-L2Find the Reward HackBrowser lab
  3. M5-L3Balance Reward TermsBrowser tool

Module 6 · Complete

Deep and Continuous-Action Reinforcement Learning

Function approximation and policy gradients: REINFORCE, baselines and advantages, actor-critic, PPO and SAC, continuous actions, and how to diagnose unstable training.

Prerequisites: M2, M4, M5.

Completion criteria
  • A learner can compute returns and advantages, write the PPO clipped objective, tune a training run, and diagnose a diverging one from its curves.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (7)

  1. M6-T1Function Approximation and Deep RLTheory
  2. M6-T2Policy Gradients and REINFORCETheory
  3. M6-T3Baselines, Advantages and GAETheory
  4. M6-T4Actor-Critic MethodsTheory
  5. M6-T5Proximal Policy OptimizationTheory
  6. M6-T6SAC and Off-Policy LearningTheory
  7. M6-T7Training Stability and DiagnosisTheory

Labs (4)

  1. M6-L1Compute Returns and AdvantagesBrowser lab
  2. M6-L2Implement the PPO Clipped ObjectiveBrowser lab
  3. M6-L3Tune a PPO RunBrowser tool
  4. M6-L4Diagnose a Diverging Training RunBrowser tool

Module 7 · Complete

Model-Based Robot Learning

Learning how the world responds: system identification, learned dynamics and their errors, planning and model predictive control with a learned model, and Dyna-style learning.

Prerequisites: M2, M4.

Completion criteria
  • A learner can fit a dynamics model, plan with it, and tell when model error makes the plan unsafe.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M7-T1Why Learn a Model?Theory
  2. M7-T2System IdentificationTheory
  3. M7-T3Learned Dynamics and Model ErrorTheory
  4. M7-T4Planning with a Model: Random Shooting and CEMTheory
  5. M7-T5Model Predictive Control with Learned ModelsTheory
  6. M7-T6Dyna and Model-Based Reinforcement LearningTheory

Labs (3)

  1. M7-L1Fit the DynamicsBrowser lab
  2. M7-L2Plan with Random ShootingBrowser lab
  3. M7-L3When the Model LiesBrowser lab

Module 8 · Complete

Offline and Data-Efficient Learning

Learning from data that were already collected: dataset coverage and quality, extrapolation error, conservative methods, replay buffers and how to get the most from limited robot experience.

Prerequisites: M3, M4, M6.

Completion criteria
  • A learner can build a replay buffer, measure dataset coverage, and compare an offline policy with an online one while explaining the gap.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (5)

  1. M8-T1Learning from Fixed DatasetsTheory
  2. M8-T2Dataset Coverage and QualityTheory
  3. M8-T3Extrapolation Error in Offline RLTheory
  4. M8-T4Conservative and Implicit MethodsTheory
  5. M8-T5Replay Buffers and Data EfficiencyTheory

Labs (3)

  1. M8-L1Build a Replay BufferBrowser lab
  2. M8-L2Find the Coverage GapBrowser lab
  3. M8-L3Compare Online and Offline PoliciesBrowser lab

Module 9 · Complete

Learning for Manipulation and Locomotion

Where learning meets contact: grasping, contact-rich manipulation, legged locomotion, curricula, and combining a learned policy with a classical controller and a safety layer.

Prerequisites: M3, M6.

Completion criteria
  • A learner can design a task reward and curriculum for a simplified manipulation or locomotion task and combine a learned policy with a safety controller.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M9-T1Learning to GraspTheory
  2. M9-T2Contact-Rich ManipulationTheory
  3. M9-T3Legged Locomotion with Reinforcement LearningTheory
  4. M9-T4Hybrid Learning and Control SystemsTheory
  5. M9-T5Curriculum and Task DesignTheory
  6. M9-T6Observations and Rewards for LocomotionTheory

Labs (3)

  1. M9-L1Predict Grasp SuccessBrowser lab
  2. M9-L2Shape a Gait RewardBrowser tool
  3. M9-L3Wrap a Policy in a Safety ControllerBrowser lab

Module 10 · Complete

Sim-to-Real Transfer and Robustness

Why simulation and reality differ and what to do about it: domain randomization, system identification, robustness testing, safe deployment and monitoring, and the bridge to foundation-model policies.

Prerequisites: M6, M7, M9.

Completion criteria
  • A learner can randomize a simulator, stress-test a policy under changed physics, detect out-of-distribution states and plan a staged rollout.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (6)

  1. M10-T1The Reality GapTheory
  2. M10-T2Domain RandomizationTheory
  3. M10-T3System Identification for TransferTheory
  4. M10-T4Robustness and Stress TestingTheory
  5. M10-T5Safe Deployment and MonitoringTheory
  6. M10-T6Foundation Models as Robot Policies: A Bridge to VLATheory

Labs (4)

  1. M10-L1Randomize the SimulatorBrowser lab
  2. M10-L2Stress-Test a PolicyBrowser lab
  3. M10-L3Detect When the Policy Is Out of DistributionBrowser lab
  4. M10-L4Plan a Staged RolloutBrowser tool

Module 11 · Complete

Research Practice and Capstone

Doing robot learning honestly: baselines, ablations, seeds and statistics, reproducibility, experiment reports, and a capstone project that designs, trains, evaluates and documents a policy.

Prerequisites: M1, M3, M4, M6.

Completion criteria
  • A learner can compare two policies with statistics, document an experiment so that it can be repeated, and complete the capstone with a defended evaluation.
  • Every planned lesson is implemented with a worked numeric example, three knowledge checks with explanations, references and working navigation.
  • Every planned lab runs in the browser: its starter code fails with a teachable message, its reference solution passes on hidden scenarios, and success and failure feedback are meaningful.
  • The module includes at least one diagnose-and-fix challenge and states the simplifications of its simulation.
  • Relevant automated tests and the build pass, pages work in light and dark mode at phone and desktop width, and no known blocking defect remains.

Theory (5)

  1. M11-T1Baselines and AblationsTheory
  2. M11-T2Seeds, Variance and StatisticsTheory
  3. M11-T3Reproducibility and Experiment TrackingTheory
  4. M11-T4Writing an Experiment ReportTheory
  5. M11-T5Designing a Robot Learning ProjectTheory

Labs (2)

  1. M11-L1Compare Two Policies StatisticallyBrowser lab
  2. M11-L2The Learning CapstoneBrowser lab