M8-T3: Verification and validation

Learning objectives

After this lesson, you should be able to:

• Distinguish verification from validation with UAV examples.
• Choose test, analysis, inspection or demonstration for a requirement and justify it.
• Grade evidence strength and state the credibility of simulation results.
• Generate a verification plan with method, level, environment and pass criterion per requirement.

Conceptual explanation

Established The four verification methods (test, analysis, inspection, demonstration) are standard in systems engineering (for example in the NASA and INCOSE handbooks). Credibility assessment of simulation follows the idea in the NASA-STD-7009 family. The concrete plan structure is a common practical choice.

Verification: confirmation, through objective evidence, that a requirement is met. Validation: confirmation that the system fulfils its intended use in the intended environment. A UAV can verify (“position error ≤ 2 m p95 in the lab motion-capture arena”) yet fail validation if the real environment (long featureless corridors, vibration, lighting) was not represented.

The four methods:

Engineering intuition

Verification is internal consistency (building the thing right); validation is external truth (building the right thing). Mistakes come in both kinds: the code can be wrong while the requirement is right (caught by verification), or the code can perfectly implement a requirement that nobody should have written (only validation catches that, usually in the field, and expensively).

The strength of evidence differs by method and by environment. A simulation can be a fine verification vehicle for logic, and a weak one for sensor behaviour unless the sensor models have been validated against real data. State the credibility: what was the simulation validated against, over what range, and what does it not model? An honest limitation statement is worth more than an overstated pass.

Mathematical formulation

Evidence accumulates in a statistical sense. For a Bernoulli success probability $p$ the Clopper-Pearson (exact) lower confidence bound after $n$ trials with $s$ successes is the value $p_{L}$ solving

For $s=n$ this reduces to $p_{L}=\alpha ^{1/n}$, which gives the success-run sample sizes of the requirements lesson. When part of the evidence comes from simulation, a simple conservative approach is to give the simulation an effective sample weight $w\in [0,\,1]$ reflecting model credibility ($w$ = 0 for an unvalidated model, rising as validation against flight data accumulates) and count only $w\,n_{\text{sim}}$ trials toward the claim, in addition to real trials. This is a heuristic for communication, not a theorem, and the weight must be justified by the validation evidence; reviewers should see both the raw and the weighted counts.

$\alpha$

significance level (1 − confidence).

$s$, $n$

successes and trials.

$w$

credibility weight of simulated evidence (judgement, documented).

Architecture and data flow

Requirement (with method and criterion) → test case (SIL, HIL, ground or flight) → execution on a stated baseline (version, parameters, calibration) → raw data and log → analysis script producing the pass/fail and the statistics → result record with the baseline ID → traceability matrix → verification report. Validation activities (field trials in representative environments, user feedback) feed back into requirement updates through change control.

Real-world UAV applications

• Building the verification plan for a GNSS-denied navigation capability.
• Reviewing whether a simulation campaign can substitute for flights.
• Deciding which requirements need flight evidence.
• Writing the limitations section of a test report.

Implementation example

This runs as shown. A verification-plan generator: requirements with a verification method and level, a check that the method is allowed for the statistical nature of the requirement, and an exact (Clopper-Pearson) lower bound with simulation credibility weighting shown separately.

import math

def clopper_pearson_lower(s, n, alpha=0.05):
    # one-sided lower bound on p after s successes in n trials, by bisection on the binomial tail
    lo, hi = 0.0, 1.0
    for _ in range(80):
        mid = (lo + hi) / 2
        tail = sum(math.comb(n, k) * mid ** k * (1 - mid) ** (n - k) for k in range(s, n + 1))
        if tail > alpha: hi = mid
        else: lo = mid
    return lo
print("zero-failure (s = n = 45): lower bound", round(clopper_pearson_lower(45, 45), 4), " alpha^(1/n) =", round(0.05 ** (1 / 45), 4))
assert abs(clopper_pearson_lower(45, 45) - 0.05 ** (1 / 45)) < 1e-6

plan = [  # id, text, method, level, statistic needed
 ('R-10', 'p95 position error <= 2 m over 5 min without GNSS', 'test', 'flight', True),
 ('R-11', 'detect VO loss within 0.5 s and land or hold', 'test', 'HIL + flight', False),
 ('R-12', 'odometry latency <= 100 ms (p99)', 'analysis', 'bench measurement + budget', True),
 ('R-13', 'camera calibration file is recorded in the flight manifest', 'inspection', 'ground', False),
 ('R-14', 'pilot can abort and take over manually', 'demonstration', 'flight', False)]
problems = [f"{i}: statistical requirement verified only by {m}" for i, t, m, l, st in plan if st and m in ('inspection', 'demonstration')]
print("plan problems:", problems or "none")
runs_real, ok_real, runs_sim, ok_sim, w = 12, 12, 400, 398, 0.1                    # 12 flights all fine; 400 simulations with 2 failures; simulation credibility weight 0.1
real_only = clopper_pearson_lower(ok_real, runs_real)
weighted = clopper_pearson_lower(ok_real + round(w * ok_sim), runs_real + round(w * runs_sim))
print(f"success probability lower bound (95 %): real flights only {real_only:.3f}; with simulation at weight {w}: {weighted:.3f} (heuristic)")
assert real_only < weighted < 1 and problems == []

Common engineering mistakes

Failure modes

Debugging approach

1. For each requirement, state the method, the level, the baseline and the sample size.
2. Compare simulated and flown results for the same scenario to measure model credibility.
3. Look for conditions in the ConOps that no test covers.
4. Have a reviewer attack the weakest piece of evidence.

Practical exercises

1. Choose the verification method for ten requirements of your capstone and justify each.
2. Compute the Clopper-Pearson lower bound for 20 successes in 21 trials.
3. List five validation questions that tests in a motion-capture arena cannot answer.
4. Write a credibility statement for a simulation you used: what it was validated against and its limits.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• NASA Systems Engineering Handbook (NASA/SP-2007-6105), verification and validation
• NASA-STD-7009: Standard for Models and Simulations (credibility assessment)
• Clopper, C. J. & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4)

What’s next