FixTheRobot Explore Robotics / Robot Learning / M11 / M11-L1

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Research Practice and Capstone · M11-L1 · Intermediate

Compare Two Policies Statistically

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, robotics office. Two training recipes for Scout are on the table: policy A is tuned in the nominal simulator, policy B is trained on randomised floors. The night-shift lead wants one sentence: "run B" or "keep A" or "we cannot say". Each recipe was trained with several seeds and every trained controller was scored on a test floor, so each policy is a short list of success rates, one per seed.

Some comparisons are easy, some are noise, one has a big gap in the means but seed scores that are all over the place, and one has a tiny gap that is consistent but would never change what the ward does. You build the three tools that turn the lists into a defensible answer: a mean with an honest interval, a permutation test, and a verdict that needs both a small p-value and a gap worth acting on. This is a toy study on the simulated Scout: no real robot data is used.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes summarise: the mean, the sample standard deviation (divide by n - 1) and the 95 % interval built with the Student-t critical value for n - 1 degrees of freedom, not the normal 1.96. Stage 2 writes permutation_p: shuffle the policy labels over the pooled scores (the shuffles are drawn for you, so the answer is deterministic), count the shuffles whose absolute difference of means is at least the observed one, and return (hits + 1) / (shuffles + 1). Stage 3 writes verdict: a winner needs p below alpha and a gap of at least the minimum effect; everything else, including a big gap with huge scatter and a tiny gap that is consistent, is "inconclusive".

    Grading is result-based on sets of seed scores, shuffles and comparisons you do not see; every score comes from a real run of the simulated Scout. The questions cover why ten seeds are not ten independent evaluations of one policy, why overlapping intervals do not by themselves mean "no difference", and why one significant result among many comparisons is weak (lesson M11-T2).

    Theory for this lab

    Research Practice and Capstone labs

    1. M11-L1: Compare Two Policies Statistically: this lab.
    2. M11-L2: The Learning Capstone

    Educational simplification: a toy study on the simulated Scout, no real robot data. One seed means one cheap training run (a controller picked from a small grid) and a success rate over 20 or 40 test starts; policies have 6 to 10 seeds (in one comparison the controllers are fixed and the seeds are evaluation seeds of 100 starts each); the permutation test is unpaired with only 200 shuffles; and the minimum important effect (0.05 to 0.10) is chosen by the lab, where a real project must fix it from the application before the experiment.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.