Reproducing intermittent failures

1. Engineering problem

Intermittent failures waste more engineering time than constant ones. Without knowing how often a failure happens, you cannot tell whether a fix worked or you were just lucky.

2. When is this needed?

• A failure appears once a day or once a week.
• After a fix, to confirm it.
• Planning stress tests.

3. Selection and design decisions

If a failure occurs with probability p per attempt, the chance of seeing it at least once in N attempts is 1 − (1 − p)^N, and about 3 / p attempts give 95 % confidence of seeing it (the “rule of three”). Levers: record every attempt (see the bag strategy guide), vary the suspected causes (load, temperature, network loss, timing, battery level), and stress them to raise p while testing.

4. Advantages and disadvantages

• Stress tests raise the failure rate and may create failures that never happen in service.
• More attempts cost time; automate them.
• Always-on recording costs disk and gives evidence the first time.

5. Requirements and constraints

• Each attempt is automated and logged with its conditions.
• The success criterion for a fix is stated as a number of failure-free attempts at the stressed rate.
• Every failure is captured with a recording around it.

6. Architecture

suspected cause ─▶ stress to raise failure rate ─▶ many automated attempts with recording ─▶ failure caught ─▶ fix ─▶ same stress, many attempts ─▶ none fail

7. Implementation

1. Estimate how often it happens in service.
2. Choose a stress that raises the rate and automate the attempts.
3. Record every attempt (snapshot or full).
4. Analyse the failures for the common factor.
5. After the fix, run enough attempts under the same stress (below).

import math
def trials_needed(p, confidence=0.95):
    return math.ceil(math.log(1 - confidence) / math.log(1 - p))
for p in (0.2, 0.05, 0.01, 0.001):
    print(f"failure rate {p * 100:5.1f} % per attempt: {trials_needed(p):5d} attempts to see it at least once with 95 % confidence; chance in 20 attempts {(1 - (1 - p) ** 20) * 100:5.1f} %")
print("rule of thumb: about 3 / p attempts for 95 % confidence of seeing a failure that happens with probability p")
print("stress the suspected cause (load, temperature, network loss) to raise p, then lower it again to confirm the fix")

Printed (checked by the site’s tests):

failure rate  20.0 % per attempt:    14 attempts to see it at least once with 95 % confidence; chance in 20 attempts  98.8 %
failure rate   5.0 % per attempt:    59 attempts to see it at least once with 95 % confidence; chance in 20 attempts  64.2 %
failure rate   1.0 % per attempt:   299 attempts to see it at least once with 95 % confidence; chance in 20 attempts  18.2 %
failure rate   0.1 % per attempt:  2995 attempts to see it at least once with 95 % confidence; chance in 20 attempts   2.0 %
rule of thumb: about 3 / p attempts for 95 % confidence of seeing a failure that happens with probability p
stress the suspected cause (load, temperature, network loss) to raise p, then lower it again to confirm the fix

Reading it: a failure with p = 5 % per attempt needs 59 attempts to be seen with 95 % confidence and is seen in 20 attempts only 64 % of the time; at 1 % it needs 299; at 0.1 % about 3 000. The consequence: 20 failure-free attempts say nothing about a 1 % failure. The numbers are standard probability.

8. Real examples

• Rare localisation jump: reproduced by injecting 20 % Wi-Fi loss while repeating a door crossing 100 times with a snapshot recorder.

9. Failure scenarios

• Declaring a fix after a few failure-free runs.
• Changing two things at once.
• No recording, so the failure is seen and not understood.

10. Troubleshooting

1. Compute the number of attempts needed before concluding.
2. Look for the common factor across the failure recordings.
3. Increase the stress if no failures appear.

11. Validation

• The calculation prints the lines shown (executed by the site tests).
• The fix is confirmed with the number of attempts the arithmetic requires at the stressed rate.

12. Engineering checklist

• Failure rate estimated
• Stress chosen
• Automated attempts
• Recording of every attempt
• Number of attempts for confidence
• Fix confirmed under stress

13. Further references

• The probability formulas are standard (binomial), and the rule of three is a standard approximation.

Related on FixTheRobot