M5-T1 — What a Reward Really Says

Learning objectives

After this lesson, you should be able to:

• Explain why an agent optimises the reward written and not the intention, and give three examples of the gap for Scout.
• Predict how the sign and size of a per-step reward change behaviour in an episodic task (finish soon against survive).
• State which reward transformations leave the optimal policy unchanged (positive scaling) and which do not (a constant added to every step when episodes can end).
• Check a reward by computation: value iteration on a small world, and trivial policies that should earn little.

The intuition

Suppose you tell a colleague, “I will pay you 5 euros for every parcel you carry through a door.” You meant: deliver parcels. A colleague who wants the money carries one parcel back and forth through the same door. You did not lie; the sentence says what it says. A learning algorithm is that colleague with no common sense, no shame and unlimited patience. It does not ask what you meant. It tries things and keeps whatever raises the number.

This is why the reward is the only specification the learner sees. Everything else you know about the task, that doors exist, that people are in corridors, that the stairwell is not a shortcut, is invisible unless some term of the reward, or the transition model, makes it matter. A designer’s job is therefore to read the reward as an adversary would: list what it pays, in every state, and ask whether the best way to collect it is the behaviour you wanted.

The figure shows the simplest version of the gap, with no cleverness at all. Scout is in an eight-cell corridor with the goal at the far end. Only the sign and size of a per-step reward change. For small or negative rewards the best policy walks to the goal. Give each step more than about half a unit and the best policy never gets there: staying put, which pays that reward every step, is worth more than ending the episode.

The mathematics

The reward hypothesis, and what it does not promise

The working assumption of reinforcement learning, which Sutton and Barto state as the reward hypothesis, is that goals and purposes can be well described as maximising the expected cumulative sum of a scalar signal. It is a modelling choice and a convenient one: one number, one objective, one algorithm. It does not say that a good scalar exists for a given intention, that you can find it by thinking for an afternoon, or that the optimiser will treat your intention and your formula as the same thing. Everything in Module 5 is about the distance between the two.

Three ways the gap shows up for Scout

These are illustrative designs, not experiments; each is made concrete in a later lesson or lab.

The pattern is the same each time: the reward is a proxy (a measurable quantity correlated with the goal), and the optimiser pushes the proxy to the place where the correlation stops holding. Goodhart’s law in everyday language is the same observation about metrics.

Scale and sign

Take a task that ends when Scout reaches the goal. Let each step pay r and the goal pay G on top of its step reward. Writing γ for the discount (lesson M4-T2), walking to the goal in n moves has value r (1 − γⁿ) / (1 − γ) + G γⁿ⁻¹ . Bumping a wall for ever, which keeps the episode alive, has value r / (1 − γ) . Compare them:

r

reward added on every step (the “step reward”), including the step that reaches the goal.

G

extra reward at the goal. With G = 10 and γ = 0.95 the threshold is r = 0.526.

n

number of moves to the goal; it cancels from the condition.

Read the sign as a statement about time. A negative step reward says “time is costly: finish soon”, and the more negative it is the more the learner will give up elsewhere to end the episode quickly, even to a worse goal. A positive step reward says “being alive is valuable”, and once it exceeds G(1 − γ)/γ the best behaviour is to avoid ending. A reward of exactly zero says only that the goal matters, which is the most honest but, as lesson M5-T3 shows, the hardest to learn from.

Terminal rewards and step costs

The goal bonus, the stairwell penalty and the step cost are three different statements. The terminal rewards say how much each way of ending is worth. The step cost says how much time is worth. The ratio decides the trade: with a stairwell penalty of −10 and a step cost of −0.1, a 20-step detour costs about 2 units and is worth taking to avoid a 30 % chance of the stairwell (an expected loss of 3); with a step cost of −1 it is not. Lesson M5-T5 turns this into a systematic account of weighted sums and constraints.

Which changes keep the optimal policy

Two reward functions are equivalent if they have the same optimal policies in every state. Two facts cover most of what a designer meets.

• Positive scaling keeps it. Replace every reward R by aR with a > 0. Every return is multiplied by a, so every ranking of policies is the same, and the values are multiplied by a. This is why the unit of a reward (euros, seconds, “points”) does not matter to the optimum. It does matter to learning: very large rewards make large updates and unstable networks (lesson M6), so scale to roughly unit size.
• Adding the same constant to every step does not keep it in general. Replace R by R + b. If the episode can end, a trajectory of T steps gains b (1 − γ^T) / (1 − γ), which depends on T: a positive b rewards longer episodes, a negative b rewards shorter ones. Only in a task that never ends does every policy gain the same b / (1 − γ), and then nothing changes.

The one additive change that is safe is not a constant but a difference of potentials, γφ(s′) − φ(s); lesson M5-T2 proves it and uses it.

A worked example

Every number comes from running RLCore.valueIteration (exact, to a tolerance of 10⁻¹²) on grids built with RLCore.gridworld. γ = 0.95 throughout.

1. Sign and size on a corridor. Eight cells in a row, Scout at the left end, the goal (+10) at the right end, seven moves apart. Bumping into the end wall leaves Scout where it is and costs one step.

The switch is at r = 10 × 0.05 / 0.95 = 0.5263; value iteration locates it by bisection at 0.5263158, the same number. A designer who adds “+0.6 per step so that Scout is rewarded for being active” has asked it never to finish.

2. Finish soon. Now a corridor with two exits: G..S....G. The near exit is three cells to the left and pays +3; the far exit is five cells to the right and pays +10. At the default step cost of −0.05 Scout goes far: Q(S, right) = 7.919 against Q(S, left) = 7.049. Make time more expensive and the answer flips. At a step cost of −4 the values are −9.952 for right and −8.703 for left, so Scout takes the near exit; the switch is at a step cost of about −3.25. Nothing about the goals changed; only the price of time.

3. Scaling is safe. Multiply every reward of the two-exit corridor (step cost −0.05) by 10. The value of the start goes from 7.919 to 79.188, exactly ten times, and the policy is the same in every cell. On the ward of lesson M4-T4 (slip 0.2, stairwell terminal, step −0.1, goal +10, stairwell −10) multiplying every reward by 5 changes the optimal action in 0 of the 23 non-terminal cells, and the largest difference between the new values and five times the old ones is 5.7 × 10⁻¹³, rounding error.

4. A constant is not safe. Add b = −4 to every reward of the two-exit corridor (so the step cost is −4.05 and the exits pay −1.05 and +5.95). Q(S, right) = −10.179 and Q(S, left) = −8.845: Scout now goes to the near exit. Add b = +1 instead and Q(S, left) = Q(S, right) = 19.000 = 0.95 / 0.05: the best thing is to bump a wall for ever. On the ward, adding b = +0.5 changes the optimal action in 3 of 23 cells (two of them now push toward a wall or away from the goal), and the true value of the resulting policy at the start is 2.382 instead of 3.314.

What the simulation simplifies. These are tiny, fully known worlds in which “the optimal policy” can be computed exactly. A real learner finds only an approximation, and the same reward can fail earlier (it never finds the goal at all, lesson M5-T3) or later (it finds a loophole nobody enumerated, lesson M5-T4). The corridor examples show the logic of the trade, not the size of the effects on a real robot.

Engineering implications and limitations

Audit a reward before training

1. Write the intention in one sentence, then write what the function pays in a table: each term, its sign, its scale, in which states it is non-zero.
2. Check the sign of the step reward against the termination: positive means “do not end” unless it is below G(1 − γ)/γ.
3. Run trivial policies and read their returns: stand still, spin in place, move at random, go straight at the goal ignoring obstacles. A trivial policy that scores well is a loophole you have found cheaply (lab M5-L2).
4. Solve the smallest version of the task exactly (value iteration on a grid) and watch the policy, not only its value.
5. Log each reward term separately during training. A total that rises while the term you care about falls is the signature of lesson M5-T4.

Prefer terminal events and measurable outcomes to hand-made proxies

“Delivered” is an event; “was probably heading toward the delivery” is a proxy. Each proxy you add is a new place for the optimiser to diverge from your intention. If you need a proxy to make learning possible, as in lesson M5-T3, use a form that provably leaves the best behaviour alone (lesson M5-T2).

Scale rewards to roughly unit size

Because positive scaling does not change the optimum, rescale so typical per-step rewards are around 0.01 to 1 and the largest event around 10. This is a numerical convenience, not a design decision. Do not use rescaling to “fix” the relative importance of two terms: their ratio is a design decision (lesson M5-T5).

Say what the reward does not cover

Write down what is left out: comfort, noise, wear, the opinion of the nurses. A short list of omissions in the design notes is honest and tells the next reader where to look when the behaviour surprises them. Constraints that must never be violated do not belong in a reward at all (lesson M5-T5).

Common misconceptions

Engineering takeaways

1. The reward is the only specification the learner sees; it optimises what is written, and the written and the intended can differ in states you never looked at.
2. The sign of the step reward states a price of time: negative means finish soon, and above G(1 − γ)/γ (0.526 for G = 10, γ = 0.95) it means do not finish.
3. Terminal rewards price the ways to end; the step cost prices time; the ratio between them decides every detour and every near-or-far choice.
4. A positive scale factor keeps the optimal policy (checked on the ward: 0 of 23 cells change); a constant added to every step does not when episodes can end (3 of 23 cells changed at +0.5).
5. Audit a reward with trivial policies and with exact value iteration on a tiny version before spending training time.

References and further reading

Sutton and Barto state the reward hypothesis; the rest is standard background and the numbers above were computed for this lesson.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (chapter 3, section on goals and rewards, where the reward hypothesis is stated).
• A. Y. Ng, D. Harada and S. Russell, “Policy invariance under reward transformations: theory and application to reward shaping”, Proceedings of the 16th International Conference on Machine Learning (ICML), 1999.
• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mané, “Concrete problems in AI safety”, arXiv:1606.06565, 2016.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: a reward that is correct but nearly silent teaches slowly. Reward shaping adds a hint that provably keeps the best behaviour, and the form it must take is a difference of potentials.

Related content

Nothing below is required to finish this lesson.

• Lab
  M5-L2: Find the Reward Hack

  Scout circles a docking station to collect "arrival" bonus forever: find the loophole and close it.