FixTheRobot Explore Robotics / Robot Learning / M1 / M1-L1

This lab works on phones, but writing code is much easier on a laptop.

Robot Learning · Foundations: Policies, Data and Evaluation · M1-L1 · Beginner

Record a Rollout

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, 14:05. Scout has driven the pharmacy corridor all morning and written every step to one long file. The training job reads it and complains: some returns look impossible, one run seems to have no ending, and a nurse swears Scout was never paid for reaching the goal.

Before anyone trains on this file, you will cut it into episodes, compute the return the robot actually earned from each step, and audit the logged rewards against the written specification. Bad data makes a bad policy, however good the learning algorithm is.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

    Hints

    What this lab teaches

    Stage 1 turns a flat log of steps into episodes using the done flag, and keeps the final run even when the log stopped before it ended. Stage 2 computes the return from every step onwards, the quantity a learning method tries to improve, and shows why the discount factor makes distant rewards count for less. Stage 3 audits the reward column against the specification (a small cost per step, a bonus for reaching the goal, a penalty for a crash), as you would before training on real logs.

    Grading is result-based: your functions are run on the practice log and on freshly generated hidden logs, and compared with the exact answer.

    Theory for this lab

    Foundations: Policies, Data and Evaluation labs

    1. M1-L1: Record a Rollout: this lab.
    2. M1-L2: Learn to Reach the Goal
    3. M1-L3: Score a Policy Fairly
    4. M1-L4: Read a Failed Training Run

    Educational simplification: one simulated task (Scout reaching a goal around a round pillar), noise-free steps, and a reward that is a plain sum of three terms. Real logs add timestamps, sensor dropouts and rewards that are learned or hand-tuned (lessons M1-T2 and M1-T3).

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.