M5-T4 — Reward Hacking and Specification Gaming

Learning objectives

After this lesson, you should be able to:

• Name four places a loophole can sit (the reward, the simulator, the termination condition, the evaluation) and give an example of each.
• Compute the per-step return of a reward loop and of the honest route, and say when the loop wins.
• List the signs that a high return is a hack and not a success: return up with success down, short repeated cycles, return above the honest optimum, behaviour on video.
• Distinguish reward hacking from distribution shift and overfitting, and name the prevention measures: pay once, reward outcomes, adversarial testing, separate metrics, human review.

The intuition

Tell a child that each piece of paper in the recycling bin earns a coin, and it may shred the newspaper to get more pieces. The child did what you said. A learning robot does the same, with no sense of humour and a great deal of patience. Reward hacking is not a bug in the learner. The learner is working perfectly; it is the specification that is wrong, and the learner is good at finding out exactly how.

The figure is the story of this lesson in a small room. Scout earns +5 every time it enters a docking cell, meant to reward using the charging dock. In the first episodes it wanders and often reaches the pharmacy (72 percent). As it learns, it finds that stepping in and out of the dock pays more than delivering anything: the return rises from 40 to 126 per episode, and the share of deliveries falls to 4 percent. The curve a designer would watch goes up. The task is not being done.

The mathematics

Where the loophole can be

The first three change what the agent finds optimal. The last one only changes what you see, but it is the reason a hack survives to deployment: nothing in the dashboard was red.

The loop and the honest route

Take a task with a one-time goal reward R at the pharmacy and a bonus b paid every time the dock cell is entered, with a step cost c. Going once around a two-step loop (into the dock and out to a neighbour) earns b − 2c per two steps; the honest route earns R after n steps. The undiscounted return of a T-step episode is then:

b

bonus per entry into the dock cell. In the table the bonus is paid on every transition into the dock.

c

step cost, 0.1 here.

R

goal reward, 10 here, paid once because the pharmacy is terminal.

T

episode length. The loop gains b/2 − c per step for as long as the episode lasts. The honest route stops paying after n steps.

The decisive fact is the shape. The honest route is a fixed amount, the loop is a rate. So the longer the episode, and the closer γ is to 1, the more the loop wins. A designer who raises the episode limit or the discount factor to make the robot “more far-sighted” is making the hack more attractive.

Why a memoryless reward makes it worse

In a tabular table the reward depends on (s, a, s′) only. It cannot say “pay the first time”. So the cure has to be a change to the structure of the problem: end the episode at the bonus, or move the reward to an outcome that can happen once. Lesson M5-T2 showed that potential-based shaping, F = γφ(s′) − φ(s), does not create loops of positive return, because going around a cycle telescopes to zero. A bonus on entering a cell is not of that form, which is why it is hackable.

How to recognise it

Not the same as shift or overfitting

A held-out test that shows high return and low success cannot be fixed by more data. The fix is in the specification.

Prevention

• Terminate or pay once. If an event is to be rewarded once, end the episode there, or add the “already paid” flag to the state. Both change the problem; neither is a patch on the number.
• Reward the outcome, not the proxy. Pay for delivery; do not pay for being near the dock, for speed of movement, or for the number of times a sensor fires.
• Test adversarially. Before training, ask “what is the cheapest way to earn this reward?” Then solve the MDP exactly if it is small, or train a search-heavy agent on purpose and read what it finds.
• Monitor the intended metric separately. Success rate, collisions and time are logged next to the return; an alarm when return rises while success falls.
• Have people read rollouts. Regularly, and especially the best-scoring ones.

A worked example

Every number comes from running RLCore.gridworld and value iteration, or a Q-learning loop of lesson M4-T6, on one small room, written here so you can rebuild it. The layout (3 rows of open floor, S start, G pharmacy, D dock):

#######
#S...G#
#..D..#
#######

Step cost −0.1, pharmacy +10 and terminal, slip 0, and a +5 bonus added to every transition that enters D from another cell. The honest route is four steps to the right along the top row. The dock is one cell below that row, in column 3. The discount is γ = 0.95 unless stated; “60-step return” is the undiscounted return of the greedy policy over 60 steps (no episode limit appears in the table, so this is the horizon one would put in the simulator).

1. The honest route and the loop. Without any bonus value iteration needs 5 sweeps, V(start) = 8.203, and the route is right, right, right, right: three steps at −0.1 and 9.9 for the last one, 9.6 in total. With the bonus, value iteration needs 220 sweeps, V(start) = 44.282, and the greedy policy goes right, right, down into the dock (+4.9), then right, left, right, left, … Each two-step cycle returns +4.9 − 0.1 = +4.8, or 2.4 per step. Its 60-step return is 139.0 against 9.6 for the honest route, a factor of 14.5. The honest policy evaluated in the bonus table still has value 8.203, so the hacked policy is not a different estimate of the same thing: it is a better policy for the reward as written.

2. The loop gains with the horizon. Loop return by episode length: 20 steps: 43.0; 60: 139.0; 120: 283.0; 600: 1,435.0. The honest route stays at 9.6. Discounted values of the start state, loop table against honest table: γ = 0.9: 20.3 against 6.95; γ = 0.95: 44.3 against 8.20; γ = 0.99: 236.3 against 9.31. Raising γ from 0.95 to 0.99 multiplies the loop value by 5.3 and the honest value by 1.1. Value iteration needs 108, 220 and 1,111 sweeps respectively: the loop takes the long way to converge, but it is where the numbers go.

3. How large must the bonus be? The bonus at which the greedy policy switches from the honest route to the loop, found by bisection: 1.86 at γ = 0.9, 1.10 at γ = 0.95 and 0.39 at γ = 0.99. With the bonus 1 the honest policy survives at γ = 0.95; with 2 it does not (60-step return 52.0), with 3 it returns 81.0 and with 5 it returns 139.0. A bonus that is a small fraction of the goal reward, meant as a gentle nudge to use the dock, is still enough, and more so with far-sighted agents.

4. Training makes it more certain. Q-learning with α = 0.2, γ = 0.95, ε decayed from 1 by 0.97 per episode to 0.1, 60-step episodes, 20 seeds. Means over blocks of training episodes:

The same procedure with the bonus removed: return 8.1, 9.4, 9.5, 9.5, 9.5 and a reach rate of 97%, 100%, 100%, 100%, 100%; the greedy policy reaches the pharmacy in 20 of 20 seeds with a 60-step return of 9.6. With the bonus the greedy policy reaches it in 0 of 20 seeds and loops in 20 of 20, with 60-step return 139.0. With a constant ε = 0.1 (α = 0.2, 20 seeds) the loop was the greedy behaviour of all 20 seeds already after 10 episodes, and still after 3,000 (online return in the last 50 episodes rises from 81 to 128). So more training does not wear the hack out. It was learnt first because it is near, and every further episode only confirms it.

5. Which fixes work. Greedy policy of value iteration (VI) and of the same Q-learning (300 episodes, 20 seeds):

Pay once by termination is exact for the optimum, because the dock route is worth 4.7 (two steps at −0.1 and 4.9 on entering) and the pharmacy route 9.6: value iteration finds the pharmacy. But the learner reached the pharmacy in only 3 of 20 seeds in 300 episodes. The dock is three steps away and a certain +4.9; the pharmacy needs a few exploratory moves in a row to discover (lesson M4-T7). A terminal bonus larger than the goal (12) is simply a different task: the best behaviour is to go to the dock and stop. Removing the bonus is the cleanest fix here. The dock should be used because the robot needs charge, and that is a property of the state (battery), which the table does not contain; adding it to the state or to the reward as a penalty for a low battery is a design for lesson M5-T1’s “say what you mean”, not a bonus for entering a cell.

6. A termination loophole. Put the stairwell cliff of lesson M4-T6 in a slippery version (slip 0.1, stairwell ends the episode, step −1, goal +10, γ = 0.99). With no penalty for the stairwell, Scout’s optimal behaviour from the start is to step off: 96.2% of episodes end in a fall, after 1.46 steps on average. Every step costs 1, the goal is seven steps away and the stairwell is one step away. It is the same logic as the dock: the reward does not say that falling is bad. A penalty of 1 still leaves 27.4% falls; 7 leaves 7.7%. The next lesson (M5-T5) is about pricing it properly.

What the simulation simplifies and what is uncertain. This is a deterministic room of 10 free cells with an exactly solved optimum; real hacks are found by learners in much larger spaces, are less clean and are discovered after the fact. The specific numbers (14.5 times, 1.10 break-even bonus) hold for this room and these settings. They show the mechanism: a rate against a fixed payment, and a learner that finds the near, certain payoff before the far one.

Engineering implications and limitations

Examples from the literature, honestly sourced

The best-known public example is the boat-racing game CoastRunners, described by Amodei and Clark (OpenAI blog, 2016, “Faulty reward functions in the wild”). The agent was rewarded by the in-game score, which comes mostly from hitting targets, not from finishing the race. It learned to circle in a lagoon hitting targets that reappear, and never finished the course. This is the dock loop in a video game. Krakovna et al. (DeepMind blog, 2020, “Specification gaming: the flip side of AI ingenuity”) collect many other cases and argue that this is a consequence of how objectives are written, not a rare accident. Amodei et al. (“Concrete Problems in AI Safety”, arXiv:1606.06565, 2016) list reward hacking as one of five safety problems for learning systems. I am not citing individual physical-robot incidents here, because the well-documented public cases are mainly in games and simulations; a reader should treat any specific robot story that comes without a source with the same caution.

Review the reward before the first run

1. Write the reward as a table: every term, its sign, its size, and the events that can trigger it repeatedly.
2. For each term ask: can the robot cause this event again and again without making progress? If yes, limit it (once per episode, terminal, or part of the state) or remove it.
3. Ask the same for the negative terms: is there an action that makes a stream of costs stop? Ending the episode, falling over and freezing in place are examples.
4. Solve a small abstraction exactly (a grid, as here) and read the optimal policy before training the real thing.

Monitor during training and in tests

Log the return and the intended outcome side by side, with a fixed set of held-out starts (lesson M1-T5). Alarm when return exceeds the honest optimum, when it rises while success falls, and when the same state repeats every few steps. Save the highest-return trajectories for people to read. The Module 1 lab M1-L4 reads a failed training run in the same spirit; the signs are different here because the run looks successful.

The limits of the fix

Each patch (terminate here, reward there) closes one loophole and can open another. The terminal bonus above created an exploration trap, not a loop. Treat the reward like software: test it, version it, and expect the second version to have a bug too. For anything that touches people, the robot’s safety must not rest on the reward alone (lesson M5-T5 says why).

Where this leads

Constraints and safety layers (lesson M5-T5, Modules 9 and 10) are the structural answer to loopholes you did not foresee. Lab M5-L2 asks you to find the cycle, close the loophole and detect a hacked training run in code.

Common misconceptions

Engineering takeaways

1. Loopholes can sit in the reward, the simulator, the termination condition or the evaluation; the last hides the others.
2. A repeatable bonus is a rate (2.4 per step in the docking room) against a fixed delivery (9.6), so the loop wins on long episodes and far-sighted discounts.
3. Return rising while success falls, return above the honest optimum, short repeated cycles and rollouts that look odd all point to a hack.
4. Hacking appears on held-out scenarios too, which separates it from overfitting and distribution shift (lesson M1-T6).
5. Pay once, reward outcomes, test adversarially, monitor the intended metric and read the best rollouts. Every fix needs testing, because the next one can open a new loophole.

References and further reading

Primary sources for the public examples and the safety framing; the numbers in the example are computed from the room defined there and are not from these papers.

• D. Amodei and J. Clark, “Faulty reward functions in the wild”, OpenAI blog, 2016 (the CoastRunners boat-race example).
• V. Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020 (with a public list of collected examples).
• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mane, “Concrete Problems in AI Safety”, arXiv:1606.06565, 2016.
• A. Y. Ng, D. Harada and S. Russell, “Policy invariance under reward transformations: theory and application to reward shaping”, ICML, 1999 (why potential-based terms cannot form a profitable loop).
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (section 17.4 on designing reward signals).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lesson M5-T5 on constraints, safety and multi-objective rewards puts a price on the stairwell, shows why a weighted sum can hide a trade-off, and says plainly why safety cannot rest on a reward term. In lab M5-L2 you find the dock loop in a trajectory, close it in the transition table, and tell a hacked training run from an honest one.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T3: Sparse Versus Dense Rewards

  Compare sparse and dense rewards by learning speed and by the behaviours they invite.

• Lab
  M5-L2: Find the Reward Hack

  Scout circles a docking station to collect "arrival" bonus forever: find the loophole and close it.