M10-T5 — Safe Deployment and Monitoring

Learning objectives

After this lesson, you should be able to:

• Plan a staged rollout from simulation to supervised trials to wider use, with a limit, a monitored signal and a rollback rule for each stage, and say what each stage is for.
• Fit a Mahalanobis-distance monitor on training observations, set its threshold from a chosen false-alarm rate, and report its detection and false-alarm rates on held-out and shifted data.
• Choose a rollback rule (k failures in the last w trials) from the policy's baseline failure rate and compute its false-rollback rate and detection delay.
• State three kinds of failure an OOD monitor cannot catch, and one case where it raises an alarm for a harmless shift.

The intuition

Think of the deployment of a new Scout policy as a sequence of gates. Each gate is a test the robot must pass before it earns the next, and each gate exists because of a specific risk, not a ritual. Run in simulation (cheap, covers the cases you thought of). Run with a human holding the stop at a low speed limit in an empty hall (finds the unmodelled behaviour, harms nobody). Run at working speed in a quiet hall with supervision (finds the interactions). Run in service with monitoring and the ability to roll back (finds the rare events, which only quantity reveals). What you do at each gate is count: incidents, interventions, and the rates of the signals you decided to watch, with a pre-agreed rule for what makes you go back.

The figure is the lesson in one picture. I fitted a Gaussian to the robot's own observations (position and velocity) during 300 simulated training episodes, and flagged an episode when three consecutive observations lay beyond the 99th percentile of the training distances. Blue bars are how often that alarm fired; orange bars are how often the episode failed. For some conditions the two agree. For others they do not. The randomised policy with a two-step delay triggers the alarm in 0.745 of episodes and never fails: the monitor correctly saw that the world had shifted, and the policy correctly did not care. The tight policy on the floor fails 0.31 of the time and the alarm fires in 0.005: the shift that kills it barely changes where it is and how fast it moves. Both mismatches are what monitoring does.

A monitor measures a property of the data (here: distance from the training cloud). The thing you care about is the policy's outcome. Staging, limits and rollback exist because the first cannot stand in for the second.

The mathematics

A Mahalanobis monitor

Take the feature vector x of one observation (here four numbers: position x, y and velocity v_x, v_y). Fit the mean and the sample covariance on N training observations:

The squared Mahalanobis distance of a new observation is

It measures distance in units of the training spread and accounts for correlation between features: two metres from the mean along a feature with a standard deviation of 0.2 m is far; along one with a standard deviation of 2 m it is not. If the data were exactly Gaussian in k dimensions, d2 would follow a chi-square distribution with k degrees of freedom, whose 99th percentile is 13.28 for k = 4 and 9.21 for k = 2. Real robot data seldom are Gaussian (a path that splits around a pillar is bimodal), so I do not use the table value: the threshold is the empirical quantile of the training scores at the chosen level.

By construction the per-observation false-alarm rate on the training data is α; on held-out data from the same distribution it should be close, and I measure how close. The robot reacts to a run, not to a single flag: an alarm needs m flags in a row (m = 3 here). Requiring a run cuts false alarms by a lot at the price of delay: at least m steps of 0.25 s.

N

Number of training observations (about 9,100 here, from 300 episodes: one observation per step, so they are correlated, not independent).

α

Chosen per-observation false-alarm rate (0.01).

m

Number of consecutive flagged observations that raise an alarm.

detection rate

Share of shifted episodes (or observations) that raise an alarm. False-alarm rate: the same measured on held-out data from the training distribution.

A rollback rule

Let f_t ∈ {0, 1} mark a failed trial. A k-of-w rule rolls back the first time a window of the last w trials contains at least k failures. With a healthy per-trial failure probability q, a window of w trials has a Binomial(w, q) number of failures, so the rule's false-rollback probability grows with q, with the number of trials in service, and as k falls. It must be chosen from the baseline failure rate q measured in the previous stage: a rule with k = 3 and w = 10 fires within 60 trials essentially for sure on a policy that fails 31% of the time. The detection delay after a real change is governed by the new failure probability q′: roughly the number of trials until k failures turn up, about k/q′ for a long window.

Limits

A limit is a bound on what the learned policy may command: a cap on the command norm (the analogue of a speed limit), an acceptable region, a bound on acceleration. It reduces how bad the outcome can be, at the price of how fast the task finishes; on Scout's 80-step episode limit, slowing down can also turn a safe run into a timeout.

A worked example

All numbers come from running the transfer study (RLCore.transferPolicy, RLCore.dynStep, RLCore.TW, the same episode logic as RLCore.transferEpisode, verified identical on a seed check) in Node. The policies are the tight theta (clear 0.3, kp 0) and the randomised one (clear 0.9, kp 1) of lesson M10-T4; episodes start from RLCore.starts(seed, 40, 'all'), seeds 201 to 205 for evaluation (200 episodes per cell) and a separate set of 300 nominal episodes (seed 101) for fitting. The scripts are scratch prototypes. Things this toy leaves out: any real sensor fault (the "shifts" are Gaussian noise, delay, gain and drag changes), the cost of a human supervisor, and everything outside Scout's four-number state.

By hand: one Mahalanobis distance

Take only the two velocity features (v_x, v_y) of the randomised policy's training data. Fitting gives mean (0.848, 0.003), variances 0.0353 and 0.1528 and a covariance of −0.0005. The inverse covariance is [[28.37, 0.09], [0.09, 6.55]]. A new reading is (0.30, 0.90): the robot is slow along x and sliding sideways, which it rarely does in the training data. The deviation is (−0.548, 0.897). Then d2 = 28.37 · 0.300 + 2 · 0.09 · (−0.548 · 0.897) + 6.55 · 0.805 = 8.51 − 0.09 + 5.27 = 13.70. Ignoring the tiny correlation, 0.300/0.0353 + 0.805/0.1528 = 8.51 + 5.27 = 13.78, nearly the same. Is that an alarm? Against the chi-square 99th percentile for two degrees of freedom (9.21), yes. Against the empirical 99th percentile of the training scores (20.39), no. The two thresholds disagree because the training velocities have heavier tails than a Gaussian. The empirical choice is the one whose false-alarm rate I can check.

Threshold and false alarms

On the full four-feature data, the 99th percentile of training d2 is 22.56, against 13.28 for a Gaussian. Measured on 2,998 fresh nominal observations: the empirical threshold flags 1.3% (target 1.0%), the chi-square threshold 3.4% (target 1.0%). At the episode level, the alarm (3 consecutive flags) fired in 0 of 200 held-out nominal episodes, so the false-alarm rate is below 1.9% (95% Wilson upper bound). Refitting on two other sets of 300 episodes (fit seeds 102 and 103) changed the numbers in the table by at most 0.05 in alarm rate (for example tight policy, delay 2: 0.175, 0.12, 0.10), so the monitor's own fit is noisy and matters when the rate is small.

Detection against failure

Three findings. (1) The monitor is dependable for large shifts in the quantity it measures: sensor noise of 0.5 is flagged in a third of episodes and 0.8 in 85%. (2) It is blind to the shift that matters most for the tight policy. On the floor it crashes in 31% of episodes while the monitor fires in 0.5%; at a one-step delay it fails in 53% and fires in 0%. A delay changes the relationship between command and effect, not the state the robot visits, until it has already gone wrong. (3) Where it speaks, it is not always about harm: delay 2 for the randomised policy is flagged in 74.5% of episodes (the oscillation that the delay induces pushes velocities outside the training cloud) and the policy succeeds every time.

Alarm, then what?

I connected the alarm to a fallback: after 3 consecutive flags the episode is abandoned with a stop command (the toy's version of the safe stop of lesson M9-T4; the glitch generator is RLCore.glitchy, the same one that lesson uses).

The result is sobering and it is the point. At delay 3 the monitor stopped 107 episodes and prevented none of the 88 crashes: the alarm came after the damage began, or on episodes that were succeeding. Success fell from 112 to 5. With sensor noise of 0.8 the stops did cut the randomised policy's crashes from 80 to 24, but at the price of 171 abandoned trips. The monitor is a smoke detector, not a seat belt: it tells a supervisor "look here", and what it should trigger is a human review or a slow-down, not an automatic stop on every alarm, unless a stop is cheap. A monitor that blocks the success you wanted needs a measured cost, and that cost has to enter the rollout decision.

A speed limit

A cap of 0.7 roughly halves the tight policy's crashes (56 to 25, success 144 to 173) and slows the randomised policy so that 11 of 200 trips time out; at 0.5 almost every trip times out. Safety purchased with time, bounded by the episode limit. A speed limit is a good first-stage setting (little can go wrong at 0.5 m/s in an empty hall) and a poor way to cure a policy.

A rollback rule

I simulated 300 deployments of the randomised policy on the floor: 60 healthy trials, then an incident (the delay creeps up from 1 to 4 steps, which drops the failure rate from about 0 to 0.24: 152 of 200 episodes succeed at floor delay 4), then 80 more trials. Each deployment is a different random start sequence and noise stream.

Stricter rules wait longer: at a post-incident failure probability of 0.24 each trial before rollback is a 24% chance of a failure, so a median delay of 8 trials (rule 2 in 10) means about two failures expected before the rule fires (8 × 0.24 = 1.9); the 3-in-10 rule costs about three. The same rules on the tight policy fire constantly: its healthy failure rate on the floor is 0.31, and in 60 healthy trials the 3-in-10, 2-in-10 and 4-in-20 rules all rolled back in 300 of 300 deployments (5 in 10: 81%). Count rules need a policy with a low baseline failure rate; a policy with a 31% baseline fails the stress test before it reaches the rollout.

Engineering implications and limitations

The staged plan

Designing each piece

• Size each stage by what you need to learn. Zero incidents in n trials only bounds the rate: the 95% upper limit is 27.8% for 10 trials, 11.4% for 30, 3.7% for 100 and 1.3% for 300 (Wilson). To show an incident rate below 1 in 100 you need hundreds of trials; hence stages are separated by quantity as well as by risk.
• Set thresholds from measured baselines. The OOD threshold from held-out training data, the rollback rule from the previous stage's failure rate. A threshold guessed from nothing either fires constantly or never.
• Fit the monitor on the data the deployed policy produces, not on a generic dataset: the tight policy's distribution and the randomised policy's differ. Refit it when the policy changes, and keep the fitting data and seed with the policy version (lesson M11 on reproducibility).
• Choose features the failure can show in. The four state features missed the delay failure. Candidate additions are the command-effect residual (predicted against observed next state, from lesson M7-T3), actuator saturation, loop timing, and the intervention rate; none of these is tested here.
• Expect false alarms and decide the response in advance. At 1% per observation and 20 observations per second, a naive alarm fires every five seconds. Requiring runs, using a different threshold per stage, and routing alarms to review rather than to an automatic stop are the usual ways to keep a monitor from being turned off.

What monitoring cannot catch

• Failures that live in the policy's response, not in the input. The tight policy on the floor crashed 31% of the time and the OOD monitor fired in 0.5% of episodes: the observations looked like training; the actuator response did not. A model-residual or a rate-of-failure monitor sees this, a distance in state space does not.
• Harmless shifts, which cost trust and trips. Delay 2 on the randomised policy raised the alarm in 74.5% of episodes and the policy never failed. A monitor that cries wolf on harmless shifts is switched off.
• Damage that has already begun. At delay 3, 107 stops prevented none of the 88 crashes: an alarm that comes after the cause is a log entry. Safety layers that act on the command (limits, the wrapper of lesson M9-T4) act earlier than monitors on the data.
• Anything you did not put in the feature vector. This is the largest group, and the reason a monitor is one layer in a stack with hardware limits, an emergency stop and a human supervisor.

What is simplified

The toy has four state features, a Gaussian model, perfect hindsight about what a failure is and shifts I generated myself. Learned-density, ensemble and reconstruction-error scores exist and behave differently; I did not test them.

Common misconceptions

Engineering takeaways

1. A staged rollout limits the cost of the first surprise: simulation and stress tests, shadow mode, supervised slow trials, supervised working speed, then monitored service, each with a limit, a watched signal and a pre-agreed rollback rule.
2. A Mahalanobis monitor with an empirical threshold had a 1.3% per-observation false-alarm rate (3.4% with the Gaussian table value) and 0 false alarms in 200 held-out episodes, flagged 85% of episodes at sensor noise 0.8, but flagged 0.5% of the tight policy's floor episodes in which it crashed 31% of the time.
3. Detection is not harm: the same monitor raised the alarm in 74.5% of episodes of a randomised policy that never failed at delay 2, and stopping on every alarm turned 112 successes at delay 3 into 5 without preventing a crash.
4. Rollback rules must be tuned to the baseline failure rate: with a post-incident failure rate of 0.24, "2 failures in 10" rolled back after a median of 8 trials, and the same rules fired constantly on a policy whose healthy failure rate was 0.31.
5. This is a toy study with no real robot data; monitoring is one layer among limits, a wrapper, a rehearsed rollback and a human, and what it cannot see (response changes, slow drift, rare events) is where the staged rollout must do the work.

References and further reading

The Mahalanobis distance is classical multivariate statistics; the use of distance scores to detect out-of-distribution inputs in learned systems is a standard idea, and I cite one paper that proposes it for deep networks, not as evidence for the numbers here, which are my own toy prototype.

• P. C. Mahalanobis, “On the generalised distance in statistics”, Proceedings of the National Institute of Sciences of India 2(1), 1936.
• K. Lee, K. Lee, H. Lee and J. Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”, Advances in Neural Information Processing Systems (NeurIPS), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Everything so far has been about a policy of a few numbers, trained in simulation, with a handful of sensors. Lesson M10-T6 looks at the other direction the field is moving: large pretrained models that take camera images and a sentence and output robot actions, and how they connect to everything in this track, and to what remains unproven.

Related content

Nothing below is required to finish this lesson.

• Before
  M10-T4: Robustness and Stress Testing

  Design tests that change friction, mass, noise, delay and actuator response, and report a robustness curve.

• Lab
  M10-L3: Detect When the Policy Is Out of Distribution

  Write a monitor that stops Scout when its observations leave the training data, without stopping it for no reason.

• Lab
  M10-L4: Plan a Staged Rollout

  Choose the stages, limits and rollback triggers for a first deployment and see which risks each one removes.