Robot Learning · Deep and Continuous-Action Reinforcement Learning · M6-L1 · Advanced
Compute Returns and Advantages
Experiment log · Scout at Meridian General Hospital
Experiment log, Meridian General Hospital, logistics corridor. Scout's new policy-gradient trainer runs, but its learning curve is strange: episodes that end in a crash seem to make the approach to the pillar look attractive, and episodes that merely run out of time are punished as if they had failed. The training logs are flat lists of rewards and value estimates from several episodes written one after another.
You will write the three pieces that turn such a batch into a learning signal: returns that stop at each episode boundary, advantages that treat a goal or a crash differently from a time-limit cut, and the two statistics (normalised advantages and explained variance) that the update and the logs depend on. Every batch is made of real rollouts in the Scout hall: a noisy expert, a reckless straight-line driver that hits the pillar, and episodes stopped early by the step limit.
Your task
Lab complete: post-mortem
You have passed every stage. Now connect it back to the theory and move on:
Output
What Python can I use?
Plain Python with math and heapq. Your code runs in a background worker: an endless loop is stopped after a few seconds and the page stays usable. It runs in your browser and is never sent anywhere.