FixTheRobot Explore Robotics / Robot Learning / M8 / M8-L1

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Offline and Data-Efficient Learning · M8-L1 · Intermediate

Build a Replay Buffer

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, east corridor. Scout runs the night-shift supply route and records every step it takes. Out of thousands of uneventful metres there are a handful of collisions with a linen cart, and the learning team has noticed something odd: a policy that avoided the cart last week now clips it again.

The replay memory is the suspect. It holds a fixed number of steps, and once it is full it must decide what to throw away and what to hand to the learner next. You will write the three pieces that decide that: the buffer that overwrites its oldest entry, the draw that picks a minibatch, and the prioritised draw that replays surprising steps more often, together with the weights that keep that from biasing the learner.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 writes a ring buffer and shows, on a log with a few early collisions, how a first-in-first-out window forgets rare events once the capacity is exceeded, next to a reservoir sample and a buffer that keeps high-priority steps. Stage 2 turns uniform random numbers into minibatch indices and exposes the off-by-one that makes the newest step unreachable. Stage 3 writes the sampling distribution and the importance weights of prioritised replay.

    Grading is result-based and exact: your buffer contents, pointer, indices, probabilities and weights are compared with a reference on logs, capacities and priorities you do not see. Random numbers are supplied from outside, because the lab's Python has no random module.

    Theory for this lab

    Offline and Data-Efficient Learning labs

    1. M8-L1: Build a Replay Buffer: this lab.
    2. M8-L2: Find the Coverage Gap
    3. M8-L3: Compare Online and Offline Policies

    Educational simplification: logs of a few hundred steps and buffers of a few dozen entries instead of the hundreds of thousands used in practice, a plain list instead of a sum tree for priorities, and retention curves that illustrate which events survive rather than benchmark how well a learner trains.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.