M8-T17: System acceptance criteria

Learning objectives

After this lesson, you should be able to:

• Write acceptance criteria with metric, threshold, statistic, sample size and conditions.
• Apply criteria to recorded results and produce an accept, reject or conditional verdict.
• Handle waivers and deviations with a recorded rationale and authority.
• Freeze the baseline and protect the criteria from retrospective changes.

Conceptual explanation

Established Acceptance practice is standard in systems engineering and contracting. The example thresholds are illustrative and do not describe any real specification.

An acceptance criterion states:

• What is measured and how (the metric, the reference, the sampling).
• The threshold and the statistic (for instance 95th percentile of horizontal error over each 5-minute segment ≤ 2.0 m).
• Conditions of the test (environment, speed, duration, configuration).
• Sample size and the confidence the claim carries (see the requirements and V&V lessons).
• The decision rule including how invalid tests, outliers and failures are treated: for example no discarding of failed runs, invalid runs rerun, all criteria must pass.

The baseline (software, parameters, calibration, hardware) is frozen for the acceptance tests; a change after the tests invalidates the results of affected criteria. Results outside the criterion are not “massaged”: they lead to rejection, rework, or a waiver (a documented acceptance of a deviation with rationale, conditions and the authority’s signature, and a corresponding operational limitation).

Engineering intuition

Think of a driving test: the examiner does not decide halfway through whether the speed limit is 50 or 60. The rule is public, the test is repeatable, and the examiner applies it. If you pass on a lucky day and fail the next, the rule needs more repetitions, not different thresholds. Acceptance for a UAV navigation system is the same, with extra care because the sample is small and the consequences are physical.

Marginal results are where integrity matters. A system that meets the 95th-percentile criterion in 44 of 45 runs may or may not pass depending on the rule written in advance. If the rule said zero exceedances, it fails, and the honest responses are to fix and retest, or to document a limitation, not to reinterpret the rule.

Mathematical formulation

A pass/fail acceptance on a success probability uses the one-sided binomial test written in advance: accept if the number of failures $x$ in $n$ independent runs is at most $x_{\max}$ where $x_{\max}$ is the largest integer such that the exact probability of observing $x\le x_{\max}$ failures when the true failure rate equals the unacceptable rate $p_{\text{bad}}$ is below the consumer’s risk $\beta$:

For $p_{\text{bad}}=0.10$ and $\beta =0.10$ with $n=22$ the plan is $x_{\max}=0$ (probability of 22 straight successes at $p_{\text{bad}}$ is $0.9^{22}\approx 0.098$). The producer’s risk (rejecting a good system) is the probability of exceeding $x_{\max}$ when the true failure rate is the acceptable one, $p_{\text{good}}$; with zero allowed failures and $p_{\text{good}}=0.02$ over 22 runs it is $1-0.98^{22}\approx 36$ %, which shows why strict plans need either a better system or a larger $n$ with $x_{\max}>0$. The operating characteristic of a plan (the accept probability against the true failure rate) should be computed and shown with the criterion.

$n$, $x_{\max}$

number of runs and maximum allowed failures.

$p_{\text{bad}}$, $p_{\text{good}}$

unacceptable and acceptable failure rates.

$\beta$

consumer’s risk (accepting a bad system).

Architecture and data flow

Requirements and criteria file (frozen, signed) → test results (with baseline ID and manifest) → acceptance evaluator script (reads criteria and results, applies the decision rule, lists invalid and missing tests) → acceptance report (pass, fail, waived, with evidence links) → review board decision, waivers recorded with authority and conditions → operational limitations document (next lesson). Any change to the baseline or criteria after the freeze triggers re-evaluation of the affected criteria.

Real-world UAV applications

• Final acceptance of a GNSS-denied navigation capability.
• Supplier acceptance tests for a sensor or a payload.
• Gate reviews between test phases.
• Documenting why a deviation was accepted and under which limits.

Implementation example

This runs as shown. An acceptance evaluator: criteria with metric, threshold, statistic and plan; results with baselines; verdicts including invalid and stale results; and the sampling-plan arithmetic with its operating characteristic. The criteria and results are illustrative.

import math
from math import comb
def cdf(x, n, p): return sum(comb(n, i) * p ** i * (1 - p) ** (n - i) for i in range(x + 1))
def plan(n, p_bad, beta):
    # largest x_max with P(X <= x_max | p_bad) <= beta, or -1 if even zero failures is too probable (n too small)
    best = -1
    for x in range(n + 1):
        if cdf(x, n, p_bad) <= beta: best = x
        else: break
    return best
print("plan n = 22, p_bad = 0.10, beta = 0.10 -> x_max =", plan(22, 0.10, 0.10), "; producer risk at p_good = 0.02:", round(1 - cdf(plan(22, 0.10, 0.10), 22, 0.02), 2))
assert plan(22, 0.10, 0.10) == 0 and plan(21, 0.10, 0.10) == -1 and abs(1 - cdf(0, 22, 0.02) - 0.36) < 0.01
for n_runs in (22, 45, 80):                                                          # operating characteristic with the plan that fits n at beta = 0.10
    x = plan(n_runs, 0.10, 0.10); print(f"n = {n_runs}: x_max = {x}; P(accept) at true failure rate 0.02: {cdf(x, n_runs, 0.02):.2f}, 0.05: {cdf(x, n_runs, 0.05):.2f}, 0.10: {cdf(x, n_runs, 0.10):.2f}")

BASELINE = 'v1.4'
criteria = [dict(id='AC-1', req='R-10', text='p95 horizontal error <= 2.0 m, GNSS-denied 5 min', n_min=45, max_exceed=0),
            dict(id='AC-2', req='R-11', text='VO loss detected <= 0.5 s and safe action started', n_min=22, max_exceed=0),
            dict(id='AC-3', req='R-12', text='odometry latency p99 <= 100 ms', n_min=1, max_exceed=0)]
results = {'AC-1': dict(n=45, exceed=0, baseline='v1.4', invalid=0), 'AC-2': dict(n=22, exceed=1, baseline='v1.4', invalid=2), 'AC-3': dict(n=1, exceed=0, baseline='v1.3', invalid=0)}
def evaluate(criteria, results):
    verdicts = {}
    for c in criteria:
        r = results.get(c['id'])
        if r is None: verdicts[c['id']] = 'NOT RUN'
        elif r['baseline'] != BASELINE: verdicts[c['id']] = f"STALE (baseline {r['baseline']}, frozen {BASELINE})"
        elif r['n'] < c['n_min']: verdicts[c['id']] = f"INCOMPLETE ({r['n']} of {c['n_min']} runs)"
        elif r['exceed'] > c['max_exceed']: verdicts[c['id']] = f"FAIL ({r['exceed']} exceedance(s))" + (f", {r['invalid']} invalid run(s) must be rerun, not dropped" if r['invalid'] else '')
        else: verdicts[c['id']] = 'PASS'
    return verdicts
v = evaluate(criteria, results)
for k, s in v.items(): print(k, s)
decision = 'ACCEPT' if all(s == 'PASS' for s in v.values()) else 'REJECT or conditional acceptance with waivers'
print("decision:", decision)
assert v['AC-1'] == 'PASS' and v['AC-2'].startswith('FAIL') and v['AC-3'].startswith('STALE') and decision.startswith('REJECT')

Common engineering mistakes

Failure modes

Debugging approach

1. Check each result’s baseline ID against the frozen baseline.
2. Compute the operating characteristic of each plan.
3. List invalid and missing tests with reasons.
4. Review waivers for completeness: rationale, conditions, authority, limitation.

Practical exercises

1. Design a plan for $p_{\text{bad}}=0.05$ and $\beta =0.05$: how many runs for $x_{\max}=0$?
2. Compute the producer risk for $x_{\max}=1$ and $n=60$ at $p_{\text{good}}=0.01$.
3. Write acceptance criteria for the detection-time requirement with statistics.
4. Draft a waiver for a requirement that passed in daylight tests only.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• NASA Systems Engineering Handbook (NASA/SP-2007-6105), product validation and acceptance
• Clopper, C. J. & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4)
• NIST/SEMATECH e-Handbook of Statistical Methods: Lot acceptance (acceptance sampling)

What’s next