FixTheRobot Explore Robotics / Robot Learning / M10 / M10-L2

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Sim-to-Real Transfer and Robustness · M10-L2 · Intermediate

Stress-Test a Policy

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, facilities office. Two Scout controllers are candidates for the night shift: policy A, tuned in the nominal simulator, takes the short path round the pillar; policy B, trained on randomised worlds, takes a wider and slower one. Both reach the goal every time in the nominal simulator. The ward manager asks a fair question: what would make each of them fail?

The simulator can answer it by changing one thing at a time (actuator lag, actuator gain, floor drag, sensor noise, process noise) over six increasingly severe levels and counting successes. The counts arrive in batches of different sizes. You turn them into a robustness curve with honest intervals, read off the largest change each policy survives, and name the factor that breaks it first. This is a toy study on the simulated Scout: no real robot data is used.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes robustness_curve: pool the successes and trials of all batches at a level (never average batch rates) and attach a 95 % Wilson interval. Stage 2 writes worst_case: walk up the curve and return the last level whose interval lower end meets the requirement, stopping at the first failure. Stage 3 writes limiting_factor: for two policies swept over several factors, find the factor that breaks first by comparing positions in each factor's sweep, since the levels have different units.

    Grading is result-based on stress tests, policies and factor grids you do not see. The last question is about what a robustness curve does not cover (lesson M10-T4).

    Theory for this lab

    Sim-to-Real Transfer and Robustness labs

    1. M10-L1: Randomize the Simulator
    2. M10-L2: Stress-Test a Policy: this lab.
    3. M10-L3: Detect When the Policy Is Out of Distribution
    4. M10-L4: Plan a Staged Rollout

    Educational simplification: a toy transfer study on Scout with momentum, no real robot data. Factors are changed one at a time (real failures combine them), each level has about 40 episodes, the requirement is a lower interval end of 0.8, and the factor grids are chosen by the lab author, so the "first to break" depends on them.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.