Robot Learning · Reinforcement Learning Foundations · M4-L2 · Intermediate

Implement Q-Learning

Experiment log · Scout at Meridian General Hospital

Experiment log, Meridian General Hospital, west wing. The facilities team has rearranged the ward again and nobody has the new map. Scout can still move and sense which cell it is in, but it does not know where the walls are, which cells are slippery, or where the stairwell edge sits. All it gets back from a move is the next cell and a reward.

You will build the learner that works this out by trial and error: a rule for when to try something new, an update that moves one estimate toward what just happened, and a loop that runs it for a few hundred episodes. The simulator then follows the route your table recommends and scores it against the best possible route, which it computes with full knowledge of the ward.

Your task

Simulation

solution.pyLoading Python…

Output

Ctrl+Enter runs

What Python can I use?

Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.

Hints

What this lab teaches

Stage 1 writes epsilon-greedy action choice with the random numbers supplied from outside, including the tie-breaking that makes a run reproducible. Stage 2 writes the Q-learning update and shows why the terminal case matters when the table starts optimistically. Stage 3 assembles the loop with episodes, resets and a documented random stream, then plots the online return per episode against a reference run and draws the greedy route on the grid.

Grading is result-based: your table is turned into a greedy policy and the policy's exact expected return is compared with the optimum on grids you do not see, including one without slip and one with a different layout. The learning curve is a diagnostic, not the grade.

Reinforcement Learning Foundations labs

Educational simplification: grids of 24 to 32 cells, a few thousand environment steps, a constant learning rate and exploration rate, and a table instead of a function approximator. Real robots have far larger state spaces; Module 5 replaces the table with a neural network.

Implement Q-Learning

Your task

Lab complete: post-mortem

Output

Hints

What this lab teaches

Theory for this lab

Reinforcement Learning Foundations labs

Implement Q-Learning

Your task

Lab complete: post-mortem

Output

Hints Show hint Show the solution

What this lab teaches

Theory for this lab

Reinforcement Learning Foundations labs

Get the next lab

Hints