FixTheRobot Explore Robotics / Robot Learning / M5 / M5-L2

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Reward Design · M5-L2 · Intermediate

Find the Reward Hack

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, east wing. To encourage Scout to top up its battery on long rounds, the facilities team added a small arrival bonus at the docking station: a few points every time Scout enters the dock cell. The training report that came back is the best anyone has seen. The average return has tripled. Nobody has received a delivery for two days.

You will read the logs like an investigator. First you find the loop Scout is stuck in from its recorded traces, then you close the loophole in the reward table and check with value iteration that Scout now goes to the pharmacy, and finally you write the monitor that would have caught it: a rule that compares the return with what honest delivery could ever pay.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes a cycle detector for recorded traces: the loop has to repeat to the end of the trace, at least twice, and pay a positive reward per period; wall bumps and single steps back are repeated cells but not hacks. Stage 2 rewrites a transition table so that the arrival bonus is no longer paid, and grades the result by what value iteration does on your table in wards you do not see: it must reach the pharmacy by a near-shortest route without a loop, and nothing else in the table may change. Stage 3 writes the monitor: a run is hacked when its return is strictly above the honest optimum while its success rate is strictly below one half.

    Grading is result-based and uses deterministic wards so that traces repeat exactly. The honest tables and reference answers stay in JavaScript.

    Theory for this lab

    Reward Design labs

    1. M5-L1: Shape a Reward Without Changing the Task
    2. M5-L2: Find the Reward Hack: this lab.
    3. M5-L3: Balance Reward Terms

    Educational simplification: small deterministic wards, one dock, one pharmacy, a bonus large enough to be unmistakable, and a table without a clock. Real loopholes are subtler, the reward is rarely this easy to locate, and a monitor needs a measure of success that the optimiser cannot see and the designer trusts.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.