FixTheRobot Explore Robotics / Robot Learning / M8 / M8-L2

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Offline and Data-Efficient Learning · M8-L2 · Intermediate

Find the Coverage Gap

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, west wing. The night-shift log is large and tidy: Scout drove the same careful route every night, with a little random wobble from the floor. The learning team wants to train a better policy from this log alone, without sending Scout back out to try things.

Before anyone trains anything, you are asked the question that decides whether that can work: what does the log actually show? You will count the (cell, action) pairs, list the cells whose best action never appears in the data, and flag the cells where the learner has too few alternatives to compare. The simulator then trains a naive offline learner on the same log and shows where it really goes wrong.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 builds the count table of (state, action) pairs from a log of transitions. Stage 2 turns it into a coverage report for a target policy: which states need an action the data never took. Stage 3 flags the states where fewer than a chosen number of different actions appear, which is where an offline learner must extrapolate, and compares that prediction with the failures of a naive offline Q-learner trained on the same data.

    Grading is result-based and exact, on wards with different layouts and slip that you do not see. The point of the lab is the distinction between the size of a dataset and its coverage: a narrow behaviour policy produces many samples of few pairs.

    Theory for this lab

    Offline and Data-Efficient Learning labs

    1. M8-L1: Build a Replay Buffer
    2. M8-L2: Find the Coverage Gap: this lab.
    3. M8-L3: Compare Online and Offline Policies

    Educational simplification: a 25-cell ward and logs of at most 500 transitions, tabular counts instead of density or distance measures in a continuous state space, and a log generated by a nearly optimal policy with a little random noise, which real logs seldom are. The naive offline learner uses an optimistic value for unseen actions as a stand-in for function-approximator over-estimation.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.