FixTheRobot Explore Robotics / Robot Learning / M4 / M4-L3

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Reinforcement Learning Foundations · M4-L3 · Intermediate

Tune Exploration

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital. Scout is learning two new jobs from trial and error. In the recovery room the only reward is the pharmacy, and nobody has told Scout where it is; on the stairwell corridor the shortest route runs along an edge that costs 100 points to step off. The first training run in the room ended with a return of exactly zero in every episode.

You will not write any learning code in this lab. You set three numbers, epsilon, its decay and the learning rate, and the page trains ten agents with different random seeds each time you move a slider, then draws what happened: the grid with the route each agent learned, the learning curves with a band across the seeds, and the episode on which each seed first reached the pharmacy.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 is the sparse-reward room. With epsilon 0 a greedy agent never sees the reward, so it can never learn from it; you find how much random behaviour is needed, and how a decay that starts too soon switches exploration off exactly when it was needed. Stage 2 is the stairwell corridor, where the table is optimistic and the route is found easily but exploration is expensive: the return Scout earns while it learns is the return of the epsilon-greedy policy, not of the greedy route it ends up with. Stage 3 asks for one setting that works in both worlds, on hidden rooms and corridors.

    Grading is by thresholds on the exact value of each learned greedy policy (policy evaluation, not a lucky rollout) and on the online return, over ten seeds, on the example and on hidden variants of both worlds. The thresholds are this lab's own acceptance limits, not universal constants.

    Theory for this lab

    Reinforcement Learning Foundations labs

    1. M4-L1: Solve an MDP by Value Iteration
    2. M4-L2: Implement Q-Learning
    3. M4-L3: Tune Exploration: this lab.
    4. M4-L4: Debug a Q-Learning Agent

    Educational simplification: two tiny grid worlds, a fixed budget of 400 episodes, a zero-initialised Q-table, epsilon never below 0.02, ten seeds per setting, and a decay that multiplies epsilon once per episode. Real exploration schedules are tuned per task, often use other mechanisms (optimistic initialisation, bonuses, separate evaluation policies), and need far more seeds before any difference can be trusted.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.