FixTheRobot Explore Robotics / Robot Learning / M6 / M6-L2

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Deep and Continuous-Action Reinforcement Learning · M6-L2 · Advanced

Implement the PPO Clipped Objective

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, logistics corridor. The new Scout trainer reuses each batch of steps for several gradient epochs, and the first version of the loss was lifted from a forum post. Training is jittery: some batches barely change the policy, others change it so much that Scout forgets how to get around the pillar. The clip range is blamed, but nobody has checked what the loss does with a sample whose probability went the wrong way.

You will write the loss yourself in three small pieces: the clipped surrogate for each step, the gradient coefficient it passes to the policy (so you can see exactly which samples it silences), and the two diagnostics, approximate KL and the fraction outside the clip range, that a PPO log shows next to it. Every batch is 200 steps of the Scout hall under a Gaussian policy with exact log-probabilities.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes the clipped surrogate min(r A, clip(r) A) and shows, step by step against the ratio, why the minimum has to come after the multiplication by the advantage. Stage 2 turns it into the gradient coefficient the optimiser uses: a sample is silenced only when its ratio has already moved past the range in the direction its advantage wanted, and the harmful side keeps its full gradient. Stage 3 writes the two KL estimators and the fraction of ratios outside the range, and shows how they differ from the silenced fraction.

    Grading is result-based: your lists are compared with the exact JavaScript values on batches you have not seen, with different clip ranges (0.1 to 0.3) and different sizes of policy change. The plots draw every sample at its ratio so you can see which regions of the objective your function reproduces.

    Theory for this lab

    Deep and Continuous-Action Reinforcement Learning labs

    1. M6-L1: Compute Returns and Advantages
    2. M6-L2: Implement the PPO Clipped Objective: this lab.
    3. M6-L3: Tune a PPO Run
    4. M6-L4: Diagnose a Diverging Training Run

    Educational simplification: the "new" policy is a controlled perturbation of the old one (its mean is nudged toward the actions that had positive advantage, plus noise), not the output of PPO epochs; batches hold 200 steps instead of thousands; the advantages come from returns minus a noisy stand-in for a critic. The log-probabilities are the exact Gaussian ones for the Scout task.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.