11.6 — Safety and Reliable Autonomous Execution

Learning objectives

After this lesson, you should be able to:

• List open safety problems named in the literature.
• Use ensemble disagreement as an out-of-distribution signal.
• Set an alarm threshold from held-out data.
• Apply the rule of three to failure-free trials.

The one-sentence idea

Know when you do not know, and count the evidence.

Failure-free trials prove less than they seem.

From “usually works” to “can be left alone”

Phase 8 asked whether a policy works; phase 9 built safety around it. The question here is harder: what would it take to trust a learned policy to run on its own, without someone watching? A survey of foundation models in robotics lists, among its main open problems, the scarcity of robot-relevant training data, safety guarantees, uncertainty quantification and real-time execution.^[1] Gemini Robotics' authors likewise state that they discuss and address safety considerations for this class of models.^[2] For a general framing of what can go wrong, Amodei and colleagues call accidents harmful behaviour from flawed design of real-world systems and list five research problems: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration and robustness to distributional shift.^[3] Three of the five are the concerns of this course's lessons on shift, reward hacking and evaluation. This lesson adds two tools: knowing when the policy is out of its depth, and counting how much evidence reliability needs.

Knowing when you do not know

A policy returns an action for any input, including one unlike its training data. A cheap warning sign is disagreement among several models trained on the same data: inside the training region they agree, outside they diverge. An ensemble of small models fitted to different resamples of the data, then asked about familiar and unfamiliar inputs:

import numpy as np
rng = np.random.default_rng(0)

x_train = rng.uniform(-1, 1, 60)
y_train = np.sin(2 * x_train) + rng.normal(0, 0.05, 60)

models = []
for _ in range(10):
    idx = rng.integers(0, len(x_train), len(x_train))                 # a bootstrap resample
    models.append(np.polyfit(x_train[idx], y_train[idx], 5))

def disagreement(x):
    preds = np.array([np.polyval(m, x) for m in models])
    return float(preds.std())

for x in (0.0, 0.5, 0.9, 1.3, 2.0, 3.0):
    print(f"input {x:>4}: disagreement {disagreement(x):10.4f}   ({'inside' if abs(x) <= 1 else 'outside'} the training range)")

input  0.0: disagreement     0.0133   (inside the training range)
input  0.5: disagreement     0.0131   (inside the training range)
input  0.9: disagreement     0.0464   (inside the training range)
input  1.3: disagreement     0.4730   (outside the training range)
input  2.0: disagreement     4.5091   (outside the training range)
input  3.0: disagreement    33.9688   (outside the training range)

Toy model. Small polynomial models and a one-dimensional input. Real systems use ensembles of networks or other uncertainty estimates, with the same logic.

Disagreement stays small inside the training range (0.01 to 0.05) and explodes outside (0.47 at 1.3, 34 at 3.0). A monitor that watches it can slow the robot or ask for help when the number rises (lesson 9.7).

Setting the alarm threshold from data

How high is “too high”? Choose the threshold from data you trust: the score distribution on held-out in-distribution inputs. If you accept a 5% false-alarm rate, set the threshold at the 95th percentile of those scores. Then count how often unfamiliar inputs trigger it:

import numpy as np
rng = np.random.default_rng(1)

score_in = np.abs(rng.normal(0, 1.0, 2000))                   # alarm scores on familiar held-out inputs
threshold = np.quantile(score_in, 0.95)

for name, shift in (("familiar inputs", 0.0), ("slightly unfamiliar", 1.0), ("clearly unfamiliar", 3.0)):
    scores = np.abs(rng.normal(shift, 1.0, 2000))
    print(f"{name:<20} alarm rate {np.mean(scores > threshold) * 100:5.1f}%")

familiar inputs      alarm rate   4.4%
slightly unfamiliar  alarm rate  16.0%
clearly unfamiliar   alarm rate  84.6%

The false-alarm rate on familiar inputs is close to the 5% we chose (4.4% in this run), and the detection rate climbs with how different the inputs are: 16% for slightly unfamiliar ones and 85% for clearly unfamiliar ones. There is no free lunch: a lower false-alarm rate means missing more of the slightly unfamiliar cases, and alarms that fire too often get ignored or switched off.

How much evidence does reliability need?

Suppose a policy completes 100 tasks with no failure. Is its failure rate below 1%? Not necessarily. With zero failures in n independent trials, an approximate 95% upper bound on the true failure rate is 3/n (the “rule of three”). So demonstrating a failure rate below 1 in 1,000 takes about 3,000 failure-free trials:

for n in (30, 100, 300, 1000, 3000):
    print(f"{n:>5} trials without any failure -> failure rate could still be as high as {3 / n * 100:5.2f}% (95% bound)")

   30 trials without any failure -> failure rate could still be as high as 10.00% (95% bound)
  100 trials without any failure -> failure rate could still be as high as  3.00% (95% bound)
  300 trials without any failure -> failure rate could still be as high as  1.00% (95% bound)
 1000 trials without any failure -> failure rate could still be as high as  0.30% (95% bound)
 3000 trials without any failure -> failure rate could still be as high as  0.10% (95% bound)

The numbers show why autonomous operation is hard to certify from tests alone: each extra “nine” of reliability costs ten times more trials. This is also why standards-based safety relies on separately engineered protective functions with their own verification (for example the monitored-stop and speed-and-separation functions described for collaborative operation, via summaries of ISO 10218 and ISO/TS 15066^[4]) rather than on the average behaviour of a learned policy.

What a trustworthy deployment looks like today

Put together: a learned policy inside a bounded envelope (limits, speed and force bounds, lesson 9.4); independent monitors and tested fallbacks (9.7); an out-of-distribution alarm with a threshold set from data (above); staged rollout with gates (10.7); logging of every rejection and alarm so that failures are studied, not forgotten; and honest statements about what was verified and under which conditions. None of this makes the policy itself provably safe. It makes the system safe enough for a defined task and setting, which is the claim you can actually support.

Common misconceptions

Engineering takeaways

1. Ensemble disagreement flags unfamiliar inputs.
2. Thresholds come from held-out data.
3. Rule of three: 3/n.
4. Claim safety for the system and its setting.

Be sure how unsure you are.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Last lesson: the open research problems.

Back to the VLA roadmap