M4-T2 — Returns and Discounting

Learning objectives

After this lesson, you should be able to:

• Compute the discounted return G_t of a reward sequence and its returns to go with the recursion G_t = r_t + γ G_t+1.
• Bound an infinite-horizon return by r_max / (1 − γ) and explain why γ < 1 makes continuing tasks well defined.
• Choose γ from a task horizon and a control rate, using the effective horizon 1 / (1 − γ).
• Describe how γ changes optimal values and decisions, and why reward scale and offset also matter.

The intuition

Suppose the pharmacy door pays +10, but it is nine steps away and every step costs 1. Is the trip worth it? That depends on how the future is weighed against the present. If tomorrow’s reward counts as much as today’s, the +10 clearly beats the nine small costs. If the future is almost ignored, the costs are all that the robot can see, and the trip looks pointless.

The discount factor γ sets that exchange rate: a reward k steps from now is worth γ^k of the same reward now. The sum of weights over an infinite future is 1 / (1 − γ), so γ = 0.9 behaves like a ten-step horizon and γ = 0.99 like a hundred.

The figure shows the return to go at each cell along the same nine-step route, for three discounts. Near the goal all three agree: 9 is 9. At the start they disagree wildly: with γ = 0.5 the goal is worth almost nothing from nine steps away (about −1.96 in total), with γ = 0.99 it is clearly a good place to head (+0.58). The shape of that curve is the value function of the next lesson.

The mathematics

The discounted return

For an episode that ends at step T, the return from step t is

For a continuing task the sum has no last term:

G_t

Return from step t (the “return to go”)

γ

Discount factor; 0 ≤ γ < 1 for continuing tasks, γ ≤ 1 allowed only if every episode ends

r_max

The largest absolute reward the task can pay in one step

H_eff

Effective horizon, 1 / (1 − γ) steps

Convergence bound

If every reward is bounded by r_max in absolute value, the infinite sum is a geometric series that is bounded too:

With γ = 1 the bound is infinite, and a continuing task can have an infinite return (a robot that finds a reward loop collects it forever). That is the technical reason for γ < 1. The bound is also a guide to scale: a policy evaluated at γ = 0.99 can legitimately have values up to a hundred times the largest single reward. The truncated tail is small: the part of the return beyond step H is at most γ^H r_max / (1 − γ), so a long finite horizon approximates an infinite one.

Undiscounted versus discounted episodic returns

For an episode that is guaranteed to end, γ = 1 is legitimate and gives the total reward: the number a user cares about. A γ < 1 is then a design choice that shapes learning (shorter effective horizon, faster and less noisy credit assignment, a preference for earlier reward). It also changes which policy is optimal, because it changes the exchange rate between a cost now and a reward later.

Choosing γ for a task

Rule of thumb: pick γ so that 1 / (1 − γ) steps covers the time scale over which consequences of an action matter, then check that the step cost (or time penalty) times that horizon is not larger than the goal reward you want to be visible.

Reward scale and offset

Multiplying every reward by c > 0 multiplies every return and value by c and leaves the optimal policy unchanged; it does change the size of the numbers a learner works with, and with function approximation that matters (a reward of 1000 can destabilise a network; Module 6). Adding a constant to every reward is not harmless: in an episodic task it changes how much staying alive is worth. If you add +1 to every step, long episodes become more rewarding than short ones, and an agent may avoid the goal to keep collecting it. The reward is a price list; an offset changes prices.

A worked example

Returns to go by hand. On the stairwell corridor (4 × 8, deterministic, step reward −1, goal +10, so the last step pays +9) the edge route has nine steps: eight rewards of −1 and a last reward of 9. With γ = 0.9, work backwards: G₈ = 9; G₇ = −1 + 0.9 × 9 = 7.1; G₆ = −1 + 0.9 × 7.1 = 5.39; and so on down to G₀ = −1.821. The undiscounted return is 8 × (−1) + 9 = 1. The plot above has all three discounts; the values are 0.579 (γ = 0.99), −1.821 (0.9) and −1.957 (0.5) at the start, computed with RLCore.returnsToGo and confirmed by value iteration.

The safe route goes up three rows, across seven columns and down three: thirteen steps. At γ = 0.5 the two routes differ by 0.04: the agent is practically indifferent, because the 10 at the end is discounted by 0.5⁸ = 0.004 from the start. At γ = 0.99 the gap is nearly 4, so the choice is clear. An agent that is nearly indifferent is easily disturbed by noise in learning.

Does γ change the route? On this deterministic corridor, no: value iteration chooses up, seven right, down (the edge route) at γ = 0.5, 0.9 and 0.99, and it converges in 11 sweeps each time. The reason is that there is no randomness to be afraid of: stepping next to the stairwell is safe when moves are exact. What γ changes is the margin and the values. Even with a step cost of 0 instead of −1 the route is the same, and the start value is 9.227 at γ = 0.99 (10 × 0.99⁸): the agent is then pulled towards the goal by the discount alone, since earlier arrival is worth more. Behaviour changes only when risk enters, as in the next paragraph.

A case where γ does change behaviour. In the ward with slip 0.2 and a −10 stairwell edge, the optimal action in the cell (3,7) depends on γ. That cell is beside the edge: moving down has a 0.1 chance of slipping left onto it. At γ = 0.95 the best action is to move down (Q = 6.975 against 6.738 for pushing right into the wall). At γ = 0.99 the best action flips to pushing right into the wall (Q = 7.827 against 7.476): waiting costs only −0.1 a step and the long horizon makes it worth trading time for safety, relying on the side-slips to carry Scout down. The start value rises with γ too: −0.196 (γ = 0.5), 1.551 (0.9), 3.314 (0.95) and 6.075 (0.99). That the optimal policy sometimes means “wait for a lucky slip” is not a bug; it is what the reward and the discount say.

The bound in practice. In the ward the largest one-step reward is 9.9 (arrival), so the bound is 99 at γ = 0.9, 198 at γ = 0.95 and 990 at γ = 0.99. The true optimal value at the start is 3.3 at 0.95, a factor 60 below the bound: the bound is safe, not tight. The tail beyond step H falls under 0.01 only for H ≥ 193 steps at γ = 0.95. Fortunately nothing in the ward lasts that long under the optimal policy (about 12.5 steps on average in a 20,000-episode simulation).

Reward scale. Re-pricing the ward ten-fold (step −1, goal +100, edge −100) gives a start value of 33.137, ten times 3.3137, with the same optimal policy. Scaling the reward is free for the planner; changing the ratio of step cost to goal reward is a different task.

What the simulation simplifies. The cliff is deterministic and tiny; real tasks have noisy, delayed, partial rewards, and the right γ is usually found by trying values and watching behaviour, not by arithmetic alone.

Engineering implications and limitations

Set γ from the control rate

Tie γ to seconds, not to a habit: if the effective horizon should be 5 s and the policy runs at 10 Hz, 1 / (1 − γ) = 50, so γ = 0.98. If you change the control rate, change γ with it.

Evaluate on the undiscounted return

Train with whatever γ helps, but report success rate and the undiscounted episodic return, the quantities the robot owner cares about (lesson M1-T3). Report γ and the reward weights with each number.

Watch the margin, not just the choice

When the value gap between the two best routes is tiny (0.04 above at γ = 0.5), a learner will flip between them as noise changes its estimates. Look at action-value gaps (lesson M4-T3), not only the greedy action.

Treat time limits carefully

An episode that is cut by a time limit has not ended in a terminal state. Bootstrapping as if it had (value 0) is a bug that biases the values near the limit; treat the cut as a truncation and bootstrap from the next state.

Be careful with offsets

Before adding a constant to the reward (“to make the numbers positive”), ask whether staying alive is now worth more than finishing. If it is, the agent will learn to dawdle.

Common misconceptions

Engineering takeaways

1. The return is G_t = r_t + γ G_t+1; returns to go are computed backwards from the end of the episode.
2. With rewards bounded by r_max and γ < 1, every return is bounded by r_max / (1 − γ); the bound is safe but loose.
3. The effective horizon is about 1 / (1 − γ) steps: choose γ from the time scale of consequences and the control rate.
4. On the cliff the route stays the same for γ = 0.5, 0.9, 0.99 but the start value goes from −1.957 to 0.579 and the gap to the safe route from 0.04 to 3.96; in the ward a cell flips its best action between γ = 0.95 and 0.99.
5. Scaling all rewards by c scales the values and keeps the policy; adding a constant changes the task.

References and further reading

Chapter 3 of the textbook covers returns, discounting and episodic versus continuing tasks; the monograph treats the discounted criterion formally. Section numbers are not cited.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018, Chapter 3.
• M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: value functions, which turn the return into a number attached to each state, and the Bellman equations that relate them.

Related content

Nothing below is required to finish this lesson.

• Before
  M4-T1: Markov Decision Processes

  Write a task as states, actions, transitions, rewards and a discount, and say what the Markov property assumes.

• Lab
  M4-L4: Debug a Q-Learning Agent

  The agent's return stays flat for 5,000 episodes: find the terminal-state, discount and update bugs.