M8-T6: Test-case development

Learning objectives

After this lesson, you should be able to:

• Write a complete test case with preconditions, stimulus, expected result, criterion, repetitions and evidence.
• Design fault-injection stimuli for sensors and links and know the injection points.
• Choose pass criteria that use the right statistic and sample size.
• Implement a test case as an executable, seeded, deterministic regression test.

Conceptual explanation

Established The structure is standard in software and systems testing. The example scenario uses a toy estimator to illustrate the method: it is not a model of any particular flight stack.

A test case has seven parts:

Fault injection places a fault at a defined point in the data path: in the simulator (sensor model), between driver and consumer (a wrapper that drops, freezes or corrupts messages), in the network (delay, loss), or physically (disconnecting a cable on the ground). Choose fault types from the FMEA: dropout, freeze (stuck value), bias step, noise increase, delay, saturation, wrong frame.

Engineering intuition

The temptation after a run is to look at the plot and decide whether it “looks okay”. A written criterion removes that bias. It also protects you from the opposite problem: a result that looks bad but meets the criterion (noise in a plot) or that looks good but fails it on the tail. The “invalid” verdict matters as much as pass and fail: a run in which the reference system dropped out proves nothing and must not count toward either.

Seed everything random and record the seed with the result. A failure you cannot reproduce cannot be fixed; a failure with a seed becomes a unit test.

Mathematical formulation

For a continuous error metric the pass criterion is usually on a percentile of the error over runs or time. To verify “the 95th percentile of the error is below L” with confidence from $n$ independent run-level maxima or run-level metrics, count the runs exceeding $L$: the claim is supported if the number of exceedances $x$ is small enough that the Clopper-Pearson upper bound on the exceedance probability is below 5 %:

For $x=0$ this needs $n\ge \ln \,\alpha /\ln \,0.95$ trials (45 at $\alpha =0.10$, 59 at $\alpha =0.05$). A metric defined per sample (error at 10 Hz for 5 minutes) has strongly correlated samples: the effective sample size is much smaller than the sample count, so use a run-level statistic or a block bootstrap, not the raw count. For fault-detection requirements (“detect within 0.5 s”) the criterion is on the detection delay distribution and on the probability of detection, with a false-alarm limit measured over fault-free runs.

$\alpha$

significance level.

$x$, $n$

number of exceedances and of runs.

$L$

error threshold in the requirement.

Architecture and data flow

Test case file (YAML) → harness (SIL/HIL runner or flight card) → baseline loaded and verified → scenario with seeded fault injection → recorded logs → analysis function computing the metric → verdict logic (pass, fail, invalid) → result record with baseline ID, seed, data pointer → traceability matrix. In CI, the same harness runs a reduced set per commit and the full set per release.

Real-world UAV applications

• Verifying the GNSS-loss failsafe timing.
• Verifying drift bounds under VIO degradation.
• Regression tests that catch estimator behaviour changes.
• Fault-injection campaigns derived from the FMEA.

Implementation example

This runs as shown. An executable test case: a 1D position estimator (a Kalman filter fusing GNSS position and a drifting odometry velocity), a seeded GNSS-dropout injection, and a verdict function with a run-level p95 criterion, an invalid-run rule and the zero-failure sample size. The toy estimator stands in for the system under test.

import numpy as np, math

def run_case(seed, drop_at=30.0, T=120.0, dt=0.1, vo_bias_sd=0.01):
    rng = np.random.default_rng(seed); n = int(T / dt); x_true = 0.0; v = 1.0
    xh = np.array([0.0, v]); P = np.diag([1.0, 0.1]); F = np.array([[1, dt], [0, 1]]); Q = np.diag([1e-4, 1e-4]) * dt
    vo_bias = rng.normal(0, vo_bias_sd); err = []; ref_ok = True
    for k in range(n):
        t = k * dt; x_true += v * dt
        xh = F @ xh; P = F @ P @ F.T + Q
        if t < drop_at and k % 10 == 0:                                 # GNSS at 1 Hz, until the injected dropout
            H = np.array([[1.0, 0.0]]); S = H @ P @ H.T + 4.0; K = P @ H.T / S
            xh = xh + (K * (x_true + 2.0 * rng.standard_normal() - xh[0])).ravel(); P = (np.eye(2) - K @ H) @ P
        if k % 2 == 0:                                                  # visual odometry velocity at 5 Hz, with a constant bias
            H = np.array([[0.0, 1.0]]); S = H @ P @ H.T + 0.05 ** 2; K = P @ H.T / S
            xh = xh + (K * (v + vo_bias + 0.05 * rng.standard_normal() - xh[1])).ravel(); P = (np.eye(2) - K @ H) @ P
        if t >= drop_at: err.append(abs(xh[0] - x_true))
    return np.array(err), ref_ok
def verdict(errs, L=3.0, req_p=0.95, alpha=0.10):
    # run-level metric: the maximum error in the denied segment; criterion: <= L in at least req_p of runs (zero-failure demonstration)
    n_needed = math.ceil(math.log(alpha) / math.log(req_p)); maxima = np.array([e.max() for e in errs])
    x = int((maxima > L).sum())
    if len(errs) < n_needed: return 'INVALID (too few runs: need %d)' % n_needed, maxima
    return ('PASS' if x == 0 else 'FAIL (%d of %d runs exceeded %.1f m)' % (x, len(errs), L)), maxima
runs = [run_case(s)[0] for s in range(45)]
v1, maxima = verdict(runs)
print(f"45 runs: median of run maxima {np.median(maxima):.2f} m, worst {maxima.max():.2f} m -> {v1}")
v2, _ = verdict(runs[:20]); print("20 runs ->", v2)
assert v2.startswith('INVALID') and math.ceil(math.log(0.10) / math.log(0.95)) == 45
v3, _ = verdict([run_case(s, vo_bias_sd=0.2)[0] for s in range(45)]); print("with a larger odometry bias (0.2 m/s sd):", v3)
assert v1 == 'PASS' and v3.startswith('FAIL')

Common engineering mistakes

Failure modes

Debugging approach

1. Re-run the failing seed with logging on and plot the estimate against the reference.
2. Check the preconditions that were actually met against those specified.
3. Compare run-level metrics between baselines to spot regressions.
4. Review injected faults against the FMEA for coverage.

Practical exercises

1. Write the test case table for a VIO dropout of 2 s with a detection-time requirement of 0.5 s.
2. Compute the number of runs for 99 % at 90 % confidence with zero failures.
3. Extend the example with a 10 s GNSS spoof (a constant 5 m offset) and decide the criterion.
4. Add an invalid-run rule for missing reference data.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• Clopper, C. J. & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4)
• NASA Systems Engineering Handbook (NASA/SP-2007-6105)
• pytest documentation

What’s next