M5-T5 — Constraints, Safety and Multi-Objective Rewards

Learning objectives

After this lesson, you should be able to:

• Explain what the weights of a weighted-sum reward mean as exchange rates, and find the penalty at which the optimal route changes.
• Read a Pareto front of time against risk and say which points a weighted sum can reach.
• State a constrained MDP, write the Lagrangian, and explain how the multiplier plays the role of a penalty chosen to meet a limit.
• Say why safety cannot rest on a reward term alone, and name what else is needed (shield, limits, monitoring, review).

The intuition

Suppose a hospital porter is paid by the delivery and docked a fine for every accident. If the fine is small compared to the time saved by cutting a corner, a rational porter cuts the corner and pays the fine. Nobody is surprised: the fine is a price. A learner is that porter, with a perfect calculator. The question for the designer is therefore not “is the penalty large?” but “at what exchange rate between minutes and accidents does the behaviour change, and is the exchange rate I set on the right side of it?”

The figure shows the answer for the slippery stairwell corridor. Every dot is the optimal behaviour for one penalty. Between a penalty of 7 and 40 the behaviour hardly changes (7.7 percent of runs fall, then 5.3), even though the penalty grew by a factor of six. The zero-fall behaviour appears only at about 295, and it is a different kind of behaviour: slow, 32 steps instead of 12. The trade-off lives in the gaps between the dots, and the weighted sum picks a dot, not the gap.

The mathematics

Weighted sums

With several goals the reward is a sum r = r_time + w_fall r_fall + w_comfort r_comfort. Only the ratios of the weights matter: multiplying all of them by 2 changes the return scale, not the optimal policy. A weight is therefore an exchange rate.

c

cost of one step (0.1 on the ward, 1 on the corridor).

λ

penalty for a fall. In a probability-times-penalty sum it is the price of 100 percent of a fall.

κ

comfort penalty per change of the commanded direction, a stand-in for jerk (see below).

ΔP, ΔE

differences between two candidate behaviours, for example a short risky route and a long safe one.

Between two behaviours the one with the larger weighted total wins. So the weights set where the switch happens, and everything on either side of that point looks identical to the learner: it makes no difference whether the penalty is 1 or 20 if the route flips at 24.

Pareto fronts

A behaviour is Pareto optimal if no other behaviour is better in one measure without being worse in another. The set of such points in the (time, risk) plane is the Pareto front. Sweeping the weight of one term traces some of the front. A weighted sum with positive weights reaches only the corners of the convex hull of the achievable points, and a deterministic policy gives you corners only. Between two corner policies there is a straight line, reached by mixing them: choose the safe policy for a fraction q of episodes and the fast one for the rest. Expected values are linear in q, so a mixture reaches every point on that segment, which is why constraints can ask for exactly what a weighted sum cannot hit.

Thresholds and constrained MDPs

Instead of paying for risk, require a bound on it. A constrained MDP (Altman, 1999) has a reward and one or more costs, and asks for the best policy among those whose expected cost stays below a limit:

c_t

a cost signal separate from the reward, for example 1 on the step of a fall and 0 otherwise.

d

the allowed expected total cost, for example a fall probability of 1 percent.

λ

the Lagrange multiplier: the price of the constraint, found to meet the limit and not chosen by hand.

For a fixed λ the Lagrangian is an ordinary MDP with reward r − λ c: exactly the weighted sum. The difference is who sets λ. A designer who writes the weighted sum guesses it. A Lagrangian method raises λ when the constraint is violated and lowers it when there is slack (a dual ascent step λ ← max(0, λ + η (J_c − d))) so that the learner ends at the multiplier that satisfies the limit. Practical algorithms built on this idea exist for deep policies (for example Achiam et al., 2017); the García and Fernández survey (2015) sorts the broader field of safe reinforcement learning into methods that change the objective and methods that change the exploration. In a finite constrained MDP the optimum may need a randomised policy (Altman), which is the mixture above.

Two cautions. The limit holds in expectation: a 1 percent bound allows one fall in a hundred runs, it does not forbid the fall. And it holds for the model: if the real slip is higher, so are the falls.

Comfort: jerk and its interactions

Jerk is the rate of change of acceleration, and passengers and payloads feel it. A tabular grid has no acceleration, so the stand-in used here is a turn penalty: a cost κ for every change of the commanded direction. The state is extended with the previous action so the reward is a function of the state, as lesson M4-T1 requires. In continuous control one subtracts a smoothness penalty such as κ‖a_t − a_t−1‖² from the reward. Comfort terms interact with the others in two ways. They add to the cost of any route with turns, so the safe detour (more turns) can become relatively more expensive and the policy flips back to the risky route. And a large enough comfort weight makes standing still the cheapest policy, because doing nothing has no jerk.

Shields and safety layers

A shield sits between the learned policy and the actuators and replaces any action that could lead to a forbidden state by a safe one (Alshiekh et al., 2018; lesson M4-T7 had a one-step shield on the cliff). It does not care what the reward says. Hybrid systems of a learned policy inside a classical controller with limits are the subject of lesson M9-T4, and staged rollout and monitoring of lesson M10-T5.

A worked example

Every number is computed with RLCore.gridworld and exact value iteration; fall probability, expected steps and expected turns are then evaluated exactly (undiscounted) for the resulting policy from the start state. Worlds: ward, the 7 × 9 ward with slip 0.2, step −0.1, goal +10, the stairwell ends the episode, γ = 0.95, penalty λ varied; corridor, the 4 × 8 stairwell corridor with slip 0.1, step −1, goal +10, the stairwell ends the episode, γ = 0.99. (This corridor differs from lesson M4-T6’s deterministic one, where a fall resets Scout.)

1. A stairwell penalty sweep on the ward.

The optimal route is identical for every penalty from 0 to 20. The policy switches at λ = 24.24, where the two routes have the same value. A designer who tests −10, the same size as the goal reward, sees no change from −2 and believes the penalty “does nothing”; one who sets −10 believing it is large still has 1 run in 9 falling (about 11.1 percent). The behaviour changes only when the exchange rate passes about 242 steps of time per fall (24.24 / 0.1). Doubling or halving the step cost moves the threshold in proportion: only the ratio matters.

The slip matters too. At slip 0, a penalty of −10 gives 10 steps and no falls. At slip 0.1 the same penalty gives 11.12 steps and 5.26 percent falls. At 0.2, 12.53 and 11.11 percent. At 0.3 the optimal route is already the long one (21.56 steps, no falls). The same penalty is adequate in one ward and inadequate in the next, which is what lab M5-L3 tests.

2. The corridor and its Pareto points. Sweeping the penalty in steps of 1 from 0 to −1000 gave eight different optimal policies:

The figure plots them. Raising the penalty from 7 to 40 reduced falls by 2.4 percentage points (7.67 to 5.26) and added 0.6 steps. The remaining 5.26 percent needed λ ≈ 295 and 19.6 more steps. A penalty of 40 is already four times the goal reward, and still 1 run in 19 falls. It is the cheapest-penalty trap: the penalty grows by orders of magnitude and the risk by almost nothing, until the whole behaviour changes.

The zero-fall policy deserves a second look. Without a better action, it stays on the start cell, bumps into the left wall and waits for slip to push it up: safe, 32 steps, and a behaviour that would depend entirely on the slip being exactly what the model says. A reward term produced it correctly; nobody would deploy it.

3. A constraint instead of a price. Suppose the requirement is P(fall) ≤ 1 percent on the ward. A deterministic policy has two choices: short (11.11 percent) or long (0 percent), so it must be long, 22.81 steps. Mixing, to meet the cap exactly, is cheaper: take the long route with probability q = 1 − 0.01 / 0.1111 = 0.91 per episode, giving 21.89 expected steps and exactly 1 percent. For a cap of 5 percent the mixture q = 0.55 gives 18.19 steps against 22.81. On the corridor with a cap of 1 percent, mixing the 5.26 percent and the 0 percent policies with q = 0.81 gives 28.24 steps against 31.96. This is the constrained optimum (Altman), not reachable by any single weighted sum. Whether a randomised policy is acceptable is an engineering question: it means that on some days Scout takes the risky route on purpose.

4. Dual ascent on the ward. Cap 5 percent, start with λ = 0, step size η = 200, update λ ← max(0, λ + η (P(fall) − 0.05)), and solve the MDP at each λ:

The multiplier crosses the switching value 24.24 and the policy flips every iteration or two. The sequence of policies never settles (over 40 iterations the long policy was used in 52.5 percent of them, close to the exact mixture 55 percent). A Lagrangian method needs the average of the policies, or a small step size and a learned multiplier, to land on the constrained optimum. With a policy that is itself trained, this oscillation shows up as unstable training, which is one reason constrained methods are harder to tune than weighted sums.

5. Comfort interacts. The ward with a turn penalty κ per change of commanded direction (state extended with the last action; turns counted on the commanded actions):

At λ = 30 the safe detour has 5.5 expected turns against 1.5 for the short route. A comfort weight of 0.223 flips the optimum back to the risky route (bisection): the comfort term costs safety. The penalty needed to keep falls below 1 percent rises with κ: 24.2 at κ = 0, 29.4 at 0.2, 37.1 at 0.5 and 49.8 at 1. At κ = 5 the robot mostly waits instead of turning: 465 expected steps at λ = 10, and 102.7 at λ = 30. Smoothness bought by doing nothing is the dock loop of lesson M5-T4 in another form.

What the simulation simplifies. Two small worlds, a single hazard, exact slip, and a turn count instead of jerk. The optimisation uses discounted returns (γ 0.95, 0.99) and the reported falls and steps are undiscounted, so the switching penalty would move a little if discounting were different. The exact penalties (24.24, 295) belong to these layouts; the pattern (long plateaus, then a jump; weights that interact; a stalling optimum for large weights) is the lesson.

Engineering implications and limitations

Why safety cannot rest on a reward term alone

This section covers engineering concepts for learners; it is not a safety case, and a real robot that works near people needs a hazard analysis by qualified people under the applicable standards. The reasons a reward term, however well tuned, is not enough are structural:

• A penalty is a price, not a prohibition. The ward at λ = 20 still has 11.1 percent falls. The optimiser will pay any price that the exchange rate makes worth paying.
• Expected values hide rare events. A constraint on the average still allows the single fall that matters. Rare events are hard to see in training: a 0.1 percent failure needs thousands of episodes to observe at all.
• The reward may itself be wrong. Lesson M5-T4 showed a bonus that overrode the task; a safety term is another line of the same specification, with the same capacity for loopholes (the corridor’s zero-fall policy waits for a slip).
• The model may be wrong. The slip of 0.1 or 0.2 above was a number in a table. A penalty tuned in simulation does not carry its fall probability to a floor that is wetter (lesson M1-T6, and Module 10).
• A learned policy is approximate. Outside its training states there is no promise about what it does (lesson M3-T3).

What helps is a stack of independent layers, each of which does not rely on the layers above being right: a shield or safety layer that vetoes actions into forbidden states (Alshiekh et al., 2018) and speed, force and region limits implemented outside the learned policy (lesson M9-T4); monitoring and out-of-distribution detection with a safe fallback (lesson M10-T5); hardware stops that do not depend on software; staged rollout with people watching; and review of the specification and the logs. The learned policy and its reward are the performance layer. They may be allowed to be wrong at the layers below.

Choosing among penalties, constraints and shields

Practical advice

1. Log every term of the reward, and each safety and comfort measure, separately, per episode. Never optimise or judge a single combined number.
2. Sweep the weights, plot the front, and write down where the behaviour changes. Choose the weights from the plot, not from a guess, and re-run the sweep when the world (slip, load, floor) changes.
3. State safety requirements as limits on measured quantities (for example, fall probability below a bound on a held-out set of starts) and check them with an independent test, not with the training reward.
4. Watch for stalling: when a comfort or safety weight is raised, check that the robot still finishes the task.
5. Put forbidden events under a shield or a limit outside the learner, and keep the penalty as a secondary signal.

Common misconceptions

Engineering takeaways

1. The weights of a sum are exchange rates; only their ratios matter, and the optimal behaviour is constant between switching points (24.24 on the ward).
2. Raising a penalty can do nothing for a long range and then change the whole behaviour: on the corridor, 7 to 40 cut falls from 7.7 to 5.3 percent and 295 was needed for none.
3. A Pareto front lists the achievable trade-offs; a weighted sum reaches its corners, a constraint with a randomised policy reaches the segments between them.
4. A constrained MDP puts the limit in a constraint and gets the price from a multiplier λ; a Lagrangian method finds λ, with oscillation that needs care.
5. Comfort terms interact with safety terms and can push the optimum toward risk or toward standing still.
6. Safety cannot rest on a reward term alone: add shields, limits outside the learner, monitoring, staged rollout and review.

References and further reading

The constrained-MDP textbook, a survey of safe reinforcement learning, and primary sources for constrained policy optimisation and shielding. The numbers in the example come from the computations described there.

• E. Altman, Constrained Markov Decision Processes, Chapman and Hall/CRC, 1999.
• J. García and F. Fernández, “A Comprehensive Survey on Safe Reinforcement Learning”, Journal of Machine Learning Research 16, 1437–1480, 2015.
• J. Achiam, D. Held, A. Tamar and P. Abbeel, “Constrained Policy Optimization”, ICML, 2017.
• M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum and U. Topcu, “Safe Reinforcement Learning via Shielding”, AAAI, 2018.
• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman and D. Mane, “Concrete Problems in AI Safety”, arXiv:1606.06565, 2016 (safe exploration and avoiding side effects).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lab M5-L3 gives you sliders for time, stairwell and comfort weights, shows each setting’s route, expected time, fall probability and comfort, and asks you to find the setting that quietly trades safety for time. Module 6 then leaves the table behind and brings neural networks into reinforcement learning.

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T4: Reward Hacking and Specification Gaming

  Recognise a policy that exploits a reward and trace the loophole to a missing term or an unintended state.

• Lab
  M5-L3: Balance Reward Terms

  Choose weights for speed, collisions and comfort, and see which settings quietly trade safety for time.