M10-T4 — Robustness and Stress Testing

Learning objectives

After this lesson, you should be able to:

• Design a stress test that varies one physical or sensing factor at a time over a stated range, with the other factors fixed and a stated number of episodes per level.
• Aggregate episodes into a success rate with a 95% Wilson interval per level, pooling episodes rather than averaging rates.
• Read a robustness curve: the largest perturbation still above a required success rate, and the failure mode beyond it.
• Explain why the worst case, not the average, is the number a deployment decision needs, and write down what a robustness report must contain.

The intuition

Module 10's toy has a policy with three numbers, theta = (clear, kp, ff). clear is how far beyond the pillar's collision circle Scout swings; kp is velocity feedback. Two thetas matter here. Tight (clear 0.3, kp 0) is what a grid search picks when it is trained in the nominal simulator only: shortest path, no margin. Randomised (clear 0.9, kp 1) is what the same search picks when each training episode draws gain, drag, delay, sensor noise and process noise from ranges that cover the floor. Both reach the goal in every one of 200 nominal episodes. On the floor the tight policy reaches it in 145 of 200 (95% interval 0.66 to 0.78) and the randomised one in 200 of 200 (0.98 to 1.00).

That one floor number is the point where the story starts, not ends. The figure shows what a stress test adds: take the nominal world and change only the actuator delay, from 0 to 4 steps. The tight policy collapses the moment the delay is one step (88 of 200, 0.37 to 0.51) and never recovers. The randomised policy is unaffected through two steps, which is the range it trained on, and then falls off a cliff: 107 of 200 at three steps, 54 of 200 at four. Robustness is not a property a policy has; it is a region, and the region ends somewhere. The test finds where.

Three habits follow. Change one thing at a time so a failure has a name. Count enough episodes that an interval, not a point, is what you read. And look past the average of the curve to its worst point, because that is where the robot will be on its worst day.

The mathematics

A stress test is a family of worlds

Let w be a world (here the five numbers gain, drag, delay, obsNoise and noise of the transfer study) and w₀ the nominal one. A one-factor sweep varies a single factor f over levels λ₁ < … < λ_m while all others stay at nominal:

For each level, run n episodes with the same policy and the same list of start positions (so the levels differ only in the world) and record outcomes as success, crash or timeout. The success rate of a level pools all its episodes. If the episodes come from several seeds with n_s episodes and k_s successes each:

Averaging the per-seed rates instead would weight a seed with 10 episodes the same as one with 190, which is wrong whenever n_s differs (and the same whenever it does not, so the pooled version is the safe habit). The uncertainty comes from the 95% Wilson interval for k successes in n trials, which behaves well near 0 and 1 where the usual normal interval does not:

λ_j

The j-th level of the stressed factor, for example delay 0, 1, 2, 3, 4 steps or an observation noise of 0, 0.05, 0.1, 0.15, 0.2 (metres, and m/s on the velocity reading).

p_j, [lo_j, hi_j]

Pooled success rate at level j and its 95% Wilson interval.

s_min

The success the application requires, for example 0.90 (a requirement, written down before the test).

safe level

The largest λ_j such that every level up to and including j satisfies the requirement.

Where the curve bends

Two rules read a requirement off a curve. The point rule asks p_j ≥ s_min. The conservative rule asks lo_j ≥ s_min, so a level only passes if even the pessimistic end of its interval does. Define the safe level through the last level before the first failure, not the last level that happens to pass: with a noisy curve, a later level can pass by luck (or by an interaction, as the table below shows) and calling it safe would hide the failure in between.

Worst case against average

Given a suite of K test conditions (single-factor levels and combinations of factors) and success rates p₁ … p_K, two summaries are tempting:

The average rewards a policy that is excellent on many easy conditions; the minimum is the number that says whether any condition the robot may meet is a disaster. They answer different questions: the average estimates performance if the conditions are drawn with equal probability (rarely true, and you rarely know the probabilities); the minimum bounds performance on any condition in the suite, and says nothing about conditions outside it. A report states both, and names the condition that sets the minimum.

A worked example

Every number comes from running RLCore.transferScore(theta, world, seed, 40) in Node for seeds 1 to 5 (so 200 episodes per cell, the same 40 starts per seed in every cell), with theta = {clear 0.3, kp 0, ff 1} (tight) and {clear 0.9, kp 1, ff 1} (randomised), and the nominal world {gain 3, drag 3, delay 0, obsNoise 0, noise 0} as the base. Both thetas were produced by RLCore.trainTheta (nominal only, and with ranges gain 2 to 3.2, drag 2.8 to 3.8, delay 0 to 2, obsNoise 0 to 0.08, noise 0 to 0.04; seeds 1 to 3 gave tight = clear 0.3, kp 0 each time, and kp 2, 1, 1 for the randomised one, of which I use kp 1). The scripts are scratch prototypes. The factors are simplified stand-ins: delay is whole steps, "gain" scales the actuator response, "drag" the friction-like damping; there is no mass, no contact and no real friction.

By hand: one Wilson interval and one pooled rate

Tight policy, delay 1: 88 successes in 200. p = 0.44 and z² = 3.8416. The denominator is 1 + 3.8416/200 = 1.0192. The centre is (0.44 + 3.8416/400)/1.0192 = 0.4411. The half-width is 1.96 · √(0.44 · 0.56/200 + 3.8416/160000)/1.0192 = 1.96 · √0.001256/1.0192 = 0.0682. The interval is 0.373 to 0.509, matching the program. For a perfect cell, 200 of 200, the upper end is 1 and the lower end is 200/(200 + 3.8416) = 0.981: even a clean sweep over 200 episodes leaves up to 2% of failures unexcluded. Pooled against averaged: if one seed had 5 successes in 10 episodes and another 171 in 190, the pooled rate is 176/200 = 0.88; the mean of the two rates is (0.5 + 0.9)/2 = 0.70.

One factor at a time, 200 episodes per cell

Read the table in four ways. (1) The tight policy's problem is delay and sensor noise: at delay 1 it crashes in 112 of 200 episodes. Gain and drag changes only slow it down (29 steps at nominal, 57 at gain 0.5), so a test that varied only those two would have called it robust. (2) The randomised policy survives everything it was trained on, and then fails differently beyond the range: at delay 3 and 4 it crashes (93 and 146 of 200), and at drag twice nominal it does not crash at all but runs out of the 80-step limit (54 timeouts). The failure mode changes with the factor, which is why a report names it. (3) Randomisation moved the cliff from delay 1 to delay 3, it did not remove it: the training range stopped at delay 2. (4) Steps are a second axis: the randomised policy takes about 31 steps in the nominal world against about 29.5 for the tight one, and about 44 on the floor.

Average against worst case

A ten-condition suite: nominal; delay 1; delay 2; obsNoise 0.1; gain 0.7; drag 1.5; process noise 0.04; delay 2 with obsNoise 0.1; delay 1 with gain 0.7 and drag 1.3; the floor world.

The tight policy's average of 0.76 is not a tragedy to the eye, until you see that the conditions it fails are the ones the floor actually has. The randomised policy's worst case of 0.945 comes from a combination (delay 2 plus noise 0.1) that neither single-factor sweep singled out: 200 of 200 on delay 2 and 200 of 200 on noise 0.1, but 189 of 200 together. Combinations matter. And an interaction can run the other way: with delay 1, gain 0.7 and drag 1.3 together the tight policy scores 182 of 200, much better than delay 1 alone (88 of 200), because the slower, more damped robot makes the one-step lag matter less. A stress test finds what breaks; it does not prove that a different combination will not.

Seed variation

The tight policy at delay 1 scored 0.350, 0.425, 0.400, 0.575 and 0.450 on the five 40-episode seeds. The 40-episode interval alone is wide (about 0.15 either way), and the seed-to-seed spread (0.35 to 0.575) is of the same order, which is why I pool 200 episodes and why lesson M11-T2 treats seeds and variance in detail. All results here come from one trained theta per policy; I did not retrain with different seeds and then stress-test, so the training variance is not in these intervals.

Engineering implications and limitations

Designing the test

• Write the requirement first. "Success of at least 0.90 up to delay 2" is a test; "see how robust it is" is a plot. The requirement decides the levels and the sample size.
• Choose factors from the reality gap. Lesson M10-T1 lists the sources: dynamics (gain, friction, mass), sensing (noise, bias, dropouts), latency, contact. Sweep each by a range you can defend from measurement (system identification, lesson M10-T3), then go beyond it on purpose to find the edge.
• Include combinations. One-factor sweeps find the factor; they miss interactions, in both directions. Add a few joint conditions chosen from the floor measurements, and the floor itself as a check.
• Use common starts. Run every level on the same list of start states and seeds so that differences between levels are due to the world; a test with fresh random starts per level adds start-to-start noise to every comparison.
• Sample enough. 20 episodes (17 successes) give an interval of 0.64 to 0.95, 100 episodes (85 successes) 0.77 to 0.91, 200 episodes (170 successes) 0.79 to 0.89. A requirement of 0.90 cannot be demonstrated by 20 episodes however good the policy is: even 20 of 20 has a lower end of 0.84.
• Record the failure mode. Crash, timeout and wrong-goal need different fixes: the randomised policy crashes beyond delay 2 and times out at high drag.
• Do not tune to the test. If you change the policy until the stress suite passes, the suite becomes training data. Keep some conditions secret (as the hidden worlds of lab M10-L2 are), or report the suite as seen by the developer and run a fresh one for the decision.

What to put in the report

What the toy omits

A real robot is stressed with different instruments (shaker tables, payload changes, lighting, temperature, wear, battery state), the factors are correlated rather than independent, and the base rate of each condition in the field is unknown. Here the "reality" is itself a simulator world, so the test measures robustness to the stated family and nothing else. A policy that passes it has met the requirement on that family; whether the real floor is in the family is the same open question as in lessons M10-T1 and M10-T2. A pass is also not a safety argument: the wrapper of lesson M9-T4 and the staged rollout of lesson M10-T5 exist because tests can be wrong.

Common misconceptions

Engineering takeaways

1. A robustness curve (success against the size of one perturbation, with Wilson intervals) shows where a policy stops working; one score in one world shows nothing about that.
2. On Scout, the tight policy reached the goal in 145 of 200 floor episodes and the randomised one in 200 of 200, but the delay sweep shows the randomised policy also has a cliff (200 of 200 up to delay 2, 107 of 200 at delay 3, 54 of 200 at delay 4), where the training range ended.
3. Pool episodes across seeds, report 95% Wilson intervals, define the safe level by the first failing level, and sample enough: 20 episodes cannot demonstrate a success of 0.90.
4. Report the worst case next to the average and name its condition: on a ten-condition suite the tight policy averaged 0.76 and bottomed at 0.34; the randomised policy averaged 0.995 and bottomed at 0.945 on a combination no single-factor sweep flagged.
5. This is a toy study with no real robot data: a stress test certifies a policy against the family of worlds you built, and the gap between that family and reality is what lessons M10-T5 and M10-T6 and the field tests have to cover.

References and further reading

The Wilson interval is a standard binomial confidence interval; the sweep design, the pooled success rate and the worst-case against average discussion are textbook-level practice and my own toy prototype, not results from a paper.

• E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference”, Journal of the American Statistical Association 22(158), 1927.
• J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba and P. Abbeel, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

A stress test tells you where the policy stops working in the worlds you thought of. Lesson M10-T5 turns to the worlds you did not think of: how to stage a deployment, limit it, watch it, and detect that the data have left the training distribution.

Related content

Nothing below is required to finish this lesson.

• Before
  M10-T3: System Identification for Transfer

  Calibrate a simulator to the real robot, and compare calibrating with randomizing.

• Lab
  M10-L2: Stress-Test a Policy

  Sweep friction, mass, noise and actuator lag, and find the first change that breaks the policy.