FixTheRobot Explore Robotics / Robot Learning / M6 / M6-L3

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Deep and Continuous-Action Reinforcement Learning · M6-L3 · Intermediate

Tune a PPO Run

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital. Scout's reach task is the same one it learned by imitation in Module 1, but this time there are no demonstrations: Scout has to find the goal by trial and error, with a policy network and a critic network trained by PPO. The first training run at the lab's default settings ended with a return of -4.0 in every iteration, which is exactly what 80 steps of standing still cost.

You will not write any learning code. You set the hyper-parameters and the page runs the real trainer in your browser (720 episodes per run), then draws what it logged: success rate, return, entropy, approximate KL, the fraction of samples PPO clipped, and the critic's explained variance. Check trains three seeds and tests the mean policy on starts it never trained on.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 isolates the step size: a learning rate that is too small leaves the KL and the clip fraction at zero and the policy where it started, and you find out what the clip range does and does not do. Stage 2 spends a fixed sample budget differently: fewer, larger batches mean fewer gradient steps; more epochs reuse each batch; the initial action noise and an entropy bonus decide whether the rare first success is ever seen. Stage 3 hands over every control, including the algorithm, and asks for a setting that works on all three seeds.

    Grading is by the success rate of the deterministic mean policy on 50 held-out starts from the whole hall and 50 held-out starts near the pillar's line, on fixed training seeds. A run takes one to three seconds, so changes are applied a moment after you stop moving a slider, and every finished run is cached by its settings.

    Theory for this lab

    Deep and Continuous-Action Reinforcement Learning labs

    1. M6-L1: Compute Returns and Advantages
    2. M6-L2: Implement the PPO Clipped Objective
    3. M6-L3: Tune a PPO Run: this lab.
    4. M6-L4: Diagnose a Diverging Training Run

    Educational simplification: one small task with a single obstacle, a fixed budget of 720 episodes, a width-32 network, no vectorised environments, three fixed training seeds and thresholds that are this lab's own acceptance limits. On this sparse task PPO is strongly seed-dependent (some seeds never find the goal at any setting), so the three seeds (7, 11 and 24) were picked from seeds 1 to 24 as ones on which a small grid of reasonable settings all reach the thresholds; a real study would report many more seeds and their spread.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.