M9-T4 — Hybrid Learning and Control Systems

Learning objectives

After this lesson, you should be able to:

• Place a learned policy inside an action limit, a monitor, a fallback and a watchdog, and say which failure each layer addresses.
• Compute by hand whether a one-step monitor flags a proposed action, and what the radial fallback commands instead.
• Report crash and success with and without the wrapper over many starts and seeds, including the cost in extra steps and in goal rate.
• State what the wrapper guarantees (under an exact model, a stated invariant) and three things it cannot (wrong model, a different failure, states it cannot predict), and describe residual policies, shielding and control barrier functions in words.

The intuition

Scout’s policy is the hand-written expert of the shared Scout world (module 1), made unreliable on purpose: with probability 0.05 per step it ignores the observation and drives at full speed straight at the pillar’s centre for three steps. That is a crude stand-in for what a network does off its training distribution (a confident wrong action), and it is exactly the failure a monitor can see because it is defined by geometry. The robot’s whole safety question is "if I execute this command, where will I be a quarter of a second from now, and is that place acceptable?".

The figure shows one real episode. Without the wrapper (orange, dashed) a glitch starts at step 13, the next commands carry the robot to 0.86 m from the pillar centre (contact is at 0.95 m) and the episode ends at step 15 with a crash. With the wrapper (blue) the same policy is overridden twice (a retreat step each time, the sharp kinks in the blue line); the robot never enters the green ring (its closest approach is 1.28 m), continues, and reaches the goal after 35 steps instead of the 30 the unglitched expert needs on the same start. The five extra steps and the little zig-zag near the ring are the price.

Three ideas carry the lesson. The monitor is simple enough to reason about, which the policy is not. The guarantee it gives is conditional on a model, and the conditions are where the failures hide. And a fallback is a controller of its own that has to be designed and tested, not an afterthought.

The mathematics

The wrapper as a composition

Let the learned policy map an observation o to a proposed command u = π(o) ∈ [−1, 1]². Scout’s position p is recovered from the observation (the robot minus pillar-centre part), the pillar centre is c, the contact radius is r = 0.95 m (pillar radius 0.7 m plus robot radius 0.25 m), and one step moves the robot by v_max Δt u = 0.25 u metres. The executed command is

where L is the action limit (a rescaling of the whole vector, which preserves its direction; clipping each component separately would change the direction) and F is the monitor with its fallback:

u_max

Largest allowed command norm; 1 is the most the world accepts along one axis, smaller values slow the robot down.

m

Monitor margin in metres, added to the contact radius to allow for model error and for the time the fallback needs to bite.

g

Fallback gain: how strongly the radial retreat dominates the goal direction (2 in all experiments).

p'

One-step prediction of the next position, under the model p' = p + 0.25 u.

watchdog

A separate check on progress, not on safety: if the distance to the goal has not decreased by at least ε over K steps, hand control to a simple controller for K steps.

What it guarantees

Claim. If the model p' = p + 0.25 u is exact, the robot starts outside the contact radius r, the pillar is far from the walls, and the observation is exact, then no executed command ever brings the robot inside r. Proof sketch. At each step either u_exec passed the monitor, so the next distance is at least r + m, or it is a fallback. The blended fallback is checked by the same monitor; if it fails, the pure radial command (p − c)/|p − c| is used, which increases the distance to c by exactly 0.25 m. In every case the new distance is at least the old distance or at least r + m, so by induction it stays at least r. The argument has the shape of the invariance arguments in the safe-control literature: a set of safe states, and a choice of action that keeps the state inside the set. It says nothing else: nothing about reaching the goal, about time, or about any model other than the one written above.

The same idea in the literature, in words

• Control barrier functions (CBFs). Define a function h(x) that is non-negative exactly on the safe states (here something like |p − c|² − (r + m)²). Then, at every step, choose the action closest to the policy’s proposal among those for which h cannot decrease faster than an allowed rate. In continuous time this is a small optimisation problem solved at every control step (Ames et al. give the formulation for control-affine systems). My monitor is the crudest instance: it either accepts the proposal or replaces it, instead of finding the nearest acceptable action.
• Shielding. A shield is a monitor, synthesised from a formal specification and an abstract model of the world, that lets the learned policy act freely and overrides only the actions that would violate the specification (Alshiekh et al. for safe reinforcement learning). Same shape as the wrapper; the difference is that the safe set is computed with guarantees, in a discrete abstraction.
• Residual policies. Instead of replacing a classical controller, the learned policy adds a bounded correction to it: u = u_classical(o) + u_res(o), with |u_res| small (Johannink et al. use this for contact-rich assembly). The classical controller provides the baseline behaviour and its guarantees, at least while the residual stays small; if the learned part is removed, the robot still works. I do not test residuals here; in this world the expert is both the baseline and the thing I make faulty.

A worked example

Every number below comes from running RLCore.glitchy(RLCore.expert, rate, rand) in Node, on RLCore.starts(seed, 30, 'all'), seeds 1 to 5, so each cell is 150 episodes; the glitch random stream is SLAMCore.rng(seed + 500). The wrapper is the one in the formulation: norm limit 1, margin m = 0.2, fallback gain 2. The prototype is a scratch script, not shipped code, and the glitch fault model is the one the monitor was designed against; both facts matter for how far to trust the table.

By hand: one monitor decision

Scout is at p = (3.9, 3.0). The observation is [5.1, 0, −1.1, 0] (goal minus robot, robot minus pillar centre). The policy proposes u = (1, 0). The one-step prediction is p' = (3.9 + 0.25, 3.0) = (4.15, 3.0), which is 0.85 m from the pillar centre. Contact is at 0.95 m, so executing the proposal would be a crash: the environment confirms it (RLCore.step returns crashed). With margin 0.2 the monitor threshold is 1.15 m, so the proposal is flagged. The fallback is w = 2 · (−1, 0) + (1, 0) = (−1, 0), already unit length, so Scout is commanded to move to (3.65, 3.0), 1.35 m from the pillar centre. Three numbers a good monitor test should reproduce: 0.85 (the rejected prediction), 1.15 (the threshold), 1.35 (where the fallback lands).

How tight is the margin? The unglitched expert’s closest approach over the same 150 starts was 1.34 m (median 1.53 m, 5th percentile 1.37 m), so a margin of 0.2 (threshold 1.15 m) never interferes with the nominal policy: zero interventions at glitch rate 0.

Crashes and success, with and without the wrapper

Per-seed raw crashes at rate 0.05 were 10, 5, 10, 7, 6 out of 30 (the spread is the seed-to-seed variation of where the glitches fall), against 0 for every seed with the wrapper. The mean steps of the raw policy are lower than the wrapped one partly because crashing ends the episode early. Limiting the norm alone changed nothing (identical to raw), because the expert never commands more than norm 1; a limit of 0.5 slows the nominal expert from 30.5 to 58.6 steps with no crashes, and with the monitor at glitch 0.05 only 135 of 150 episodes finished inside the 80-step limit: safety purchased with time.

Margin and cost

At rate 0.1 a margin of 0.0, 0.1 and 0.2 gave 148, 149 and 150 successes (all zero crashes): at 0.0 and 0.1 a few episodes (2 and 1) ended without reaching the goal and I did not investigate why. The margin makes a small difference to success in this world, and none to crashes.

Where the guarantee ends

The first three rows say that the guarantee is as good as the model: with a 50% model error a margin that was fine becomes a hole (3 crashes), and a margin that covers the error restores it, at no visible cost in this world. At twice the true speed even 0.2 is not enough (7 crashes). The monitor is blind to a failure it was not designed for: a policy that freezes never approaches the pillar, so the monitor stays silent and 30 of 150 episodes run out of time; only a watchdog watching progress recovers them. What I did not test: sensing noise, delays and momentum in the monitor’s own model (the momentum dynamics of module 7 mean the robot cannot stop within a step), obstacles the monitor does not know, or a policy whose faults are not geometric.

Engineering implications and limitations

Design rules that follow

• Keep the monitor outside the learned code. The monitor must stay correct when the policy is wrong, so it should share as little as possible with it: its own copy of the geometry, its own state estimate if you can afford it, a small code path you can read in a page and test exhaustively, ideally another process.
• Order the layers. Limit first, then monitor, then fallback: a monitor that checks the unclipped command checks something the robot will not do, and a limit applied after the monitor can break the property the monitor established (two easy mistakes, both worth a test, as in lab M9-L3).
• Design the fallback as a controller. Radial retreat works near a convex pillar in an open hall. In a corridor it can drive into a wall; on a slope, stopping is not safe; with a person behind you, retreating is not safe. A fallback needs its own safe-set argument and its own tests, and the monitor should check that the fallback is itself acceptable (as in the formulation).
• Expect chatter. A threshold monitor flips between policy and fallback near the boundary, which in the figure shows as a zig-zag and in a real robot as jerky motion and wear. Hysteresis (leave the fallback only once the margin is regained), a smooth blend, or the CBF formulation that finds the nearest acceptable action reduce it.
• Pay for the margin explicitly. The margin has to cover model error, sensing error, the stopping distance and latency. Choose it from measurements (system identification, lessons M7-T2 and M10-T3, and robustness tests, lesson M10-T4), not by trying values until the table is clean, which is what a single toy sweep invites.
• Log every intervention. The intervention rate (0.0, 0.3, 0.6, 1.8 per episode in the table) is a health signal: a policy that needs the monitor often is a policy that should be retrained, and a rate that rises after an update is a regression. Lesson M10-T5 treats this as deployment monitoring.
• Pair the safety layer with a liveness layer. Safety says nothing bad happens; a watchdog (progress, heartbeat of the learned process, command timeout) says something useful keeps happening. A frozen policy is safe and useless, and you need to notice.
• Test the wrapper against faults you did not use to design it. The table reports 0 crashes against lunges at the pillar because that is what the monitor was written for. A fair test uses a different fault generator and states the number of episodes, so that zero is read as an upper bound (here 2.5% at 95% confidence).

Which approach when

What this does not replace

The wrapper is one defence in a stack that also includes hardware limits (current, force, speed), an emergency stop, physical separation and a risk assessment; it is not a certified safety function, and a formal safety argument for a product is far beyond a one-step toy monitor. Here the point is the pattern, and the habit of asking for each layer what exactly it assumes.

Common misconceptions

Engineering takeaways

1. Hybrid systems let the learned policy propose and a simple, checkable layer dispose: an action limit, a one-step monitor, a fallback, and a watchdog for stalls.
2. On Scout with a glitchy expert (150 episodes per cell), 38 of 150 episodes crashed at glitch rate 0.05 and 99 of 150 at 0.1; the wrapper with margin 0.2 gave 0 crashes in all four rates, at the cost of 30.1 to 35.6 steps and 41.5 steps respectively on average, and 0.6 and 1.8 interventions per episode.
3. The guarantee is an invariant under an exact model; at a true speed twice the modelled one the margin 0.2 wrapper still crashed 7 times in 150, and a freeze fault, which does not approach the pillar, passed straight through the monitor and was only recovered by a progress watchdog.
4. Control barrier functions, shields and residual policies are the same pattern at different levels of formality; the work is in the model and the margin, which have to come from measurement, and in the fallback, which must be designed and tested as a controller.
5. Log interventions, test against faults the monitor was not designed for, and put the wrapper into a staged rollout (lesson M10-T5).

References and further reading

Papers on control barrier functions, shielding and residual learning are the primary sources for the ideas I describe in words; the survey by Brunke and colleagues is a good map of the area. The wrapper experiments are my own toy prototype and do not reproduce any of those methods.

• A. D. Ames, X. Xu, J. W. Grizzle and P. Tabuada, “Control Barrier Function Based Quadratic Programs for Safety Critical Systems”, IEEE Transactions on Automatic Control, 2017.
• M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum and U. Topcu, “Safe Reinforcement Learning via Shielding”, AAAI Conference on Artificial Intelligence, 2018.
• T. Johannink et al., “Residual Reinforcement Learning for Robot Control”, IEEE International Conference on Robotics and Automation (ICRA), 2019.
• L. Brunke et al., “Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning”, Annual Review of Control, Robotics, and Autonomous Systems 5, 2022.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

The wrapper handles what the policy does once it is trained. Lesson M9-T5 goes back to how the policy is trained: the order and difficulty of the tasks, the design of the task itself, and how a curriculum can hide the very failure the wrapper is there to catch.

Related content

Nothing below is required to finish this lesson.

• Before
  M9-T3: Legged Locomotion with Reinforcement Learning

  Describe how a gait policy is trained in simulation, with its observations, actions and reward terms.

• Lab
  M9-L3: Wrap a Policy in a Safety Controller

  The learned policy is usually good and occasionally terrible: add limits, a monitor and a fallback.