FixTheRobot Explore Robotics / Robot Learning / M8 / M8-L3

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Offline and Data-Efficient Learning · M8-L3 · Intermediate

Compare Online and Offline Policies

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, central ward. The team has a month of Scout logs and a rule: no more test drives on the real ward until the new policy has been checked. Someone has trained a Q-learner on the log and it looks fine on the training curve. Deployed in the simulator, it stands still at the start cell and presses against the wall.

You will write the offline learner yourself, in a naive version and a version that refuses to trust actions the log never showed, add a penalty for rarely seen pairs, and then run all of them against an online learner that had the same number of transitions but was allowed to drive. The question is not which is best in general; it is what each one does in the cells where the data run out.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 implements tabular offline Q-learning from a fixed list of transitions in two modes: the naive backup, which maximises over every action, and the behaviour-constrained backup, which maximises only over actions the log shows at the next cell. Stage 2 writes the target of a count-based penalty and why it shrinks like one over the square root of the count. Stage 3 runs ten logs per setting in JavaScript and plots the exact value of each greedy policy at the start for the naive, constrained and penalty learners and for an online learner with the same transition budget.

    Grading is result-based: stages 1 and 2 compare your tables and targets exactly with the reference on wards you do not see, and stage 3 checks that the setting you choose reaches 80% of the optimal value in at least 8 of 10 runs on every hidden ward, not only on the practice example.

    Theory for this lab

    Offline and Data-Efficient Learning labs

    1. M8-L1: Build a Replay Buffer
    2. M8-L2: Find the Coverage Gap
    3. M8-L3: Compare Online and Offline Policies: this lab.

    Educational simplification: a 25-cell tabular ward; the log comes from a nearly optimal behaviour policy with random actions mixed in (real logs are usually worse); the naive learner starts unseen entries at +10 as a stand-in for the over-estimation of a function approximator; the online learner has a fixed exploration schedule and no tuning; the thresholds are this lab's own. Ten runs of a small ward are noisy, so neighbouring settings can differ.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.