FixTheRobot Explore Robotics / Robot Learning / M6 / M6-L4

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Deep and Continuous-Action Reinforcement Learning · M6-L4 · Advanced

Diagnose a Diverging Training Run

Experiment log · Scout at Meridian General Hospital

Four training runs reached the Meridian General robotics desk this week, each with the same complaint: "the return is wrong". In one the return rose and then fell off a cliff; in another it never moved from the standing-still level; in the third the critic looked asleep; in the fourth the dashboard looked like static.

You will not write any code. Each incident is a real run of the PPO trainer with one deliberately bad setting, and the log and the six diagnostic curves come from the same run. For each you name the cause from the diagnostics, choose the first parameter to change, and say what the repaired run must show to confirm you were right. After you answer, the page quotes the numbers of the repaired run.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    A bad return curve rarely says why it is bad. The diagnostics do: the approximate KL and the clip fraction say how far each update moved the policy, the entropy says whether the policy is still exploring, the critic's value loss and explained variance say whether the baseline carries information, and the scatter of the return from one iteration to the next says how noisy the batches are. The four incidents are a learning rate and clip range too aggressive (the policy learns, then collapses), an entropy term with the wrong sign (the policy freezes before it finds the goal), a critic that receives no gradient (the advantages are mostly its error), and a batch of two episodes (the gradient is mostly noise).

    Every incident asks three things: the cause, the first parameter to change and the evidence that would confirm the fix. The confirmation is not hypothetical: the explanation re-runs the trainer on the same seed with the cause repaired and quotes its numbers.

    Theory for this lab

    Deep and Continuous-Action Reinforcement Learning labs

    1. M6-L1: Compute Returns and Advantages
    2. M6-L2: Implement the PPO Clipped Objective
    3. M6-L3: Tune a PPO Run
    4. M6-L4: Diagnose a Diverging Training Run: this lab.

    Educational simplification: each incident uses one seed and one dominant, deliberately extreme cause (learning rate 0.05 with clip range 0.5, entropy coefficient -100, critic loss weight 0, two episodes per batch). Real failures overlap, develop gradually and differ from seed to seed; one repaired rerun is an illustration, not a statistical test.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.