FixTheRobot Explore Robotics / Robot Learning / M5 / M5-L1

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Reward Design · M5-L1 · Intermediate

Shape a Reward Without Changing the Task

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, east annex. The new pharmacy sits at the far end of a corridor system that took the facilities team two years to finish: dead ends, switchbacks and a pocket of storage rooms that look close to the pharmacy on the floor plan and are not. Scout is rewarded only when it arrives, and after thousands of steps of tabular Q-learning it has still not arrived once. The reward is too sparse for a lucky walk to find.

The obvious repair, paying Scout a little for getting closer, is also the classic way to break a task: a careless bonus changes which route is best, and Scout goes off to collect it. You will add progress in the one form that is known not to change the best route, check that claim with value iteration on your own table, and then build the progress measure itself, where the first obvious choice turns out to be wrong.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes the shaping term F = gamma times the next potential minus the current potential, with the terminal case that makes the terms telescope over an episode. Stage 2 applies it to every outcome of a transition table and grades the result three ways: the exact rewards, whether value iteration on your table still recommends an optimal action of the original task in every cell, and whether a Q-learning run with fixed random numbers now reaches the pharmacy within ten episodes. Stage 3 builds the potential from the walking distance around the walls using a breadth-first search, and shows what the straight-line distance does: it makes hills out of dead ends.

    Grading is result-based and runs on corridors you do not see, including slippery ones. The learner code never sees the exact tables or the optimal actions; they stay in JavaScript.

    Theory for this lab

    Reward Design labs

    1. M5-L1: Shape a Reward Without Changing the Task: this lab.
    2. M5-L2: Find the Reward Hack
    3. M5-L3: Balance Reward Terms

    Educational simplification: corridors of 50 to 60 cells, a tabular reference learner with fixed hyperparameters and a fixed random stream, and potentials that are exactly the distance to the goal. Real tasks have no exact distance to hand, shaping is rarely this clean, and policy invariance is a statement about optimal policies, not about how a learner behaves after a finite amount of training.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.