M1-T3 — Rewards, Returns and Objectives

Learning objectives

After this lesson, you should be able to:

• Write a reward function for a task and compute the return of an episode, with and without discounting.
• Use the recursion G_t = r_t + γ G_t+1 to compute returns to go.
• Explain how the discount factor sets an effective horizon of about 1/(1 − γ) steps.
• Give examples of what a reward does not say, and of a reward that differs from the intention.

The intuition

After each step the world hands the robot a reward: a small cost for every step taken, a large bonus for arriving, a penalty for a collision. The robot does not want reward the way a person wants money; the learning procedure adjusts the policy so that the reward collected over an episode grows.

The total is the return. Adding rewards exactly is the simplest return. Usually future rewards are weighed a little less than immediate ones, by a discount factor γ between 0 and 1: a reward k steps from now counts γ^k times as much. A γ near 0 makes the robot short-sighted. A γ near 1 makes it patient.

The reward is also the only way the designer speaks to the learner. Say “-0.05 per step, +10 for the goal” and Scout will find the fastest safe route. Say “+1 for each metre closer to the goal” and Scout may learn to shuffle back and forth over the same metre. The reward carries meaning only through what it makes best.

The mathematics

Reward and return

The reward at step t is a number r_t, determined by the state, the action and what followed. The return from step t, G_t, adds the rewards from t to the end of the episode, weighing step t + k by γ^k:

r_t

Reward received at step t

G_t

Return from step t onward: the “return to go”

γ

Discount factor in [0, 1]; γ = 1 is plain summing (fine for episodes that end)

T

Episode length; G_T = 0 because nothing follows the last step

The recursion gives a way to compute all returns to go in one backward pass: start at the last step and work back. It is also the seed of the Bellman equations in module 4.

Scout’s reward

The horizon effect

The weights γ^k fall off geometrically; the sum of all of them is 1/(1 − γ), and the weight at k = 1/(1 − γ) is about 1/e ≈ 0.37. So the robot effectively looks ahead about 1/(1 − γ) steps:

What a reward does not say

• How to do it. A reward gives no plan; the learner has to find one by trial.
• What you forgot. A hall with a person in it, a sleeping patient, a wet floor: not in the reward, not in the behaviour.
• The trade-offs you assumed. Is arriving 10 s sooner worth a closer pass to the pillar? The reward fixes an exchange rate whether you noticed or not.
• Safety. A −5 for a crash is a price. A learner may accept the price when the gain is larger.

A worked example

A three-step episode. Rewards [−0.05, −0.05, +10] with γ = 0.9. Work backwards: G₂ = 10; G₁ = −0.05 + 0.9 × 10 = 8.95; G₀ = −0.05 + 0.9 × 8.95 = 8.005. The returns to go are [8.005, 8.95, 10]. With γ = 1 the return is 9.90.

Scout’s real episode. From the start (1, 1) the expert takes 34 steps: 33 steps at −0.05 and a last step of −0.05 + 10 = 9.95. The same episode under different discount factors:

Why it matters. Compare three behaviours from (1, 1): reach the goal (the expert), stand still for 80 steps, and drive straight into the pillar (13 steps, from (1, 3)). With γ = 1 they score 8.30, −4.00 and −5.65: the right order. With γ = 0.9 they score −0.177, −0.500 and −1.785: the order still holds, but the gap between success and idling has shrunk from 12.3 to 0.32. With γ = 0.5, standing still scores −0.100, the same as the expert to three decimals, so a learner with that γ has almost nothing telling it to go. Discount is part of the task definition, not only a tuning knob.

Returns to go along the run. Under γ = 0.99 the returns to go at steps 0, 10, 20, 30 and 33 are 5.73, 6.87, 8.12, 9.51 and 9.95. They rise towards the goal: a good place to be is one from which a lot of reward is near. That is the idea of a value function in module 4.

What the simulation simplifies. The reward is computed from exact positions, immediately and without noise, and there is only one goal and one hazard. In practice rewards are measured, delayed, partly missing and open to disputes about what counts as arrival.

Engineering implications and limitations

Compute the return from the log

Before any training, recompute the return of a few logged episodes by hand or with a short script and check them against what the logger reports. A wrong sign or a missing terminal reward is far cheaper to find now (lab M1-L1).

Choose γ from the task

Pick γ so that 1/(1 − γ) steps cover the part of the future that matters: with a 0.25 s step, γ = 0.99 looks about 25 s ahead, γ = 0.9 only 2.5 s. For tasks that end, a γ close to 1 is common; the price is slower and noisier learning.

Report returns honestly

State the γ and the reward weights with every number. A return of 8.3 and a return of −0.18 can describe the same episode. When comparing methods, evaluate them on the undiscounted return or the success rate, whatever γ they trained with.

Read the behaviour, not the number

A high return is evidence of a good policy only if the reward means what you meant. Watch rollouts. Module 5 builds on this: shaping, sparse versus dense rewards, hacking and constraints.

Common misconceptions

Engineering takeaways

1. A reward is one number per step; the return G_t adds rewards from step t, weighted by γ^k.
2. Returns to go follow the recursion G_t = r_t + γ G_t+1, computed backwards from the last step.
3. The discount sets an effective horizon of about 1/(1 − γ) steps; too small a γ can hide a distant goal.
4. In Scout’s world, +10, −0.05 and −5 fix a price for time and for collisions; the policy trades them off exactly as written.
5. The reward written is not the intention: what it leaves out, the learner will not respect.

References and further reading

The standard textbook for returns, discounting and the reward hypothesis; a survey for robotics.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018, Chapters 1 and 3.
• J. Kober, J. A. Bagnell and J. Peters, “Reinforcement learning in robotics: A survey”, The International Journal of Robotics Research, 32(11), 2013.
• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mané, “Concrete Problems in AI Safety”, arXiv:1606.06565, 2016.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what is learned and what is fixed, and what runs on the robot rather than on a workstation.

Related content

Nothing below is required to finish this lesson.

• Before
  M1-T2: Observations, Actions and Datasets

  Describe a robot task as observations, actions and episodes, and structure a dataset from a rollout.

• Lab
  M1-L1: Record a Rollout

  Scout drives a corridor and the log is a mess: turn it into episodes, compute the return, and find the step where the reward is wrong.