FixTheRobot Explore Robotics / Robot Learning / M4 / M4-L4

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Reinforcement Learning Foundations · M4-L4 · Intermediate

Debug a Q-Learning Agent

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital. A colleague hands over three versions of Scout's tabular Q-learning agent. Each one runs without an error, each one prints plausible numbers, and none of them learns: the return per episode is flat, or the routes ignore the shortest way, or the plan at the start of the ward is arbitrary.

You will read each agent as a reviewer would: compare its table and its learning curve with a reference agent that sees exactly the same random numbers, name the symptom, find the one line responsible and fix it. The hidden tests run your function on other wards and measure the exact value of the policy your table implies.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Reinforcement-learning code fails quietly. Stage 1 is a terminal-state bug: the loop leaves the episode before it updates for the last transition, so the reward at the end never enters the table and the return curve stays flat. Stage 2 is a discount bug: gamma multiplies the reward instead of the next state's value, so values stop shrinking with distance and, in a world with no step cost, nothing prefers the short route. Stage 3 is an update-index bug: the state is advanced before the update, so the update writes into the row of the state Scout arrived in and the start cell is never learned.

    Grading is result-based and runs on freshly generated wards. The random numbers for exploration and for the transitions are generated outside your code and passed in, so the same function gives the same table every time; the policy your table implies is evaluated exactly and compared with the optimum from value iteration. Each stage also asks which symptom points to which bug.

    Theory for this lab

    Reinforcement Learning Foundations labs

    1. M4-L1: Solve an MDP by Value Iteration
    2. M4-L2: Implement Q-Learning
    3. M4-L3: Tune Exploration
    4. M4-L4: Debug a Q-Learning Agent: this lab.

    Educational simplification: tiny deterministic grid worlds, fixed hyperparameters, three planted bugs that each act alone, and one random stream shared with a reference agent so that the comparison is exact. Real bugs combine, are subtler, and rarely come with a reference run on identical random numbers; the habit of reading the symptom before editing is what carries over.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.