M8-T4: SIL, HIL and ground testing

Learning objectives

After this lesson, you should be able to:

• Describe the purpose and limits of each test level.
• Define entry criteria for moving from one level to the next.
• Design fault-injection tests per level and know where each fault can be injected.
• Build a defect-class by test-level coverage matrix and find what only flight can test.

Conceptual explanation

Established The staged approach is standard in aerospace and robotics testing. Details (which levels exist, which gates apply) depend on the programme; the table is a typical pattern, not a prescription.

Fault injection belongs at every level: at SIL and HIL by altering the simulated sensor stream or the message flow (dropouts, freezes, biases, noise bursts); on the ground by disconnecting sensors or links; in flight only for faults whose consequences are bounded by the safety design (a pilot ready to take over, a geofence).

Engineering intuition

The cost of finding a defect rises by about an order of magnitude per level. A sign error found in a unit test costs minutes; the same error found in HIL costs hours; on the ground days; in flight, possibly the vehicle. So the discipline is to ask, for every flight test: could this have been found lower? If yes, add the missing lower-level test before flying.

The reverse is also true: some things only flight reveals (vibration coupling into the IMU at specific throttle, wind gust response, lighting changes). The matrix tells you which defect classes have no cheaper level, so you plan flight time for them deliberately instead of hoping to meet them.

Mathematical formulation

A simple expected-cost model chooses how much to test at each level. If a defect class has occurrence probability $q$, and test level $l$ detects it with probability $d_{l}$ at cost $c_{l}$ (incurred whether or not it is found), and an escape to the next level costs more by a factor of $f_{l}$, then the expected cost of finding and fixing at the first detecting level is

With $d_{l}$ typically high at lower levels for logic defects and zero for environment-specific ones. The cumulative detection probability after levels 1…$L$ is $1-\Pi (1-d_{l})$; with independent detection probabilities of 0.9, 0.6 and 0.5 it is 1 − 0.1·0.4·0.5 = 0.98. Escapes that reach flight are the product of the non-detections, which is why raising the lowest-cost level’s detection rate has the biggest effect on the total. Treat the probabilities as engineering judgement informed by history; they are prioritisation aids, not measurements.

$d_{l}$

probability that level $l$ detects the defect class.

$c_{l}$

cost of executing level $l$.

$F_{l}$

cost of fixing a defect found at level $l$.

Architecture and data flow

Code change → unit tests in CI → SIL regression (many seeded scenarios) in CI → HIL bench test on a rig (nightly or per release) → ground test checklist on the vehicle → tethered flight → free flight per test card. At each level the result is stored with the baseline ID; the entry criteria for the next level read those results automatically (all required tests pass on this baseline, no open critical defects, traceability coverage complete).

Real-world UAV applications

• Planning the test campaign for a new navigation algorithm.
• Deciding what must be verified before the first outdoor flight.
• Defining the SIL regression suite that gates merges.
• Organising fault-injection tests across levels.

Implementation example

This runs as shown. A defect-class by level matrix with detection probabilities (illustrative judgement values), the cumulative detection, the classes that only flight can find, and a gate checker for entry criteria.

import numpy as np
levels = ['unit', 'SIL', 'HIL', 'ground', 'tether', 'flight']
detect = {  # defect class -> probability of detection at each level (illustrative engineering judgement)
 'frame / unit error':           [0.50, 0.90, 0.60, 0.50, 0.80, 0.90],
 'failsafe logic error':         [0.20, 0.85, 0.80, 0.60, 0.30, 0.80],
 'timing / overload on target':  [0.00, 0.00, 0.90, 0.40, 0.50, 0.70],
 'sensor driver / wiring fault': [0.00, 0.00, 0.70, 0.90, 0.80, 0.80],
 'vibration coupling into IMU':  [0.00, 0.00, 0.00, 0.60, 0.70, 0.90],
 'wind / aerodynamic limit':     [0.00, 0.10, 0.00, 0.00, 0.30, 0.90]}
for k, d in detect.items():
    cum = 1 - np.cumprod(1 - np.array(d)); first_cheap = next((levels[i] for i, c in enumerate(d) if c >= 0.5), 'never')
    print(f"{k:>28}: first level with detection >= 0.5: {first_cheap:>6}; cumulative before flight {cum[-2]:.2f}, after flight {cum[-1]:.2f}")
flight_only = [k for k, d in detect.items() if max(d[:-1]) < 0.5]
print("classes with no pre-flight level at >= 0.5 (plan flight time for these):", flight_only)
assert flight_only == ['wind / aerodynamic limit']
results = {'unit': 'pass', 'SIL': 'pass', 'HIL': 'pass', 'ground': 'fail', 'tether': 'not run'}
def gate(next_level):
    need = levels[:levels.index(next_level)]; blockers = [l for l in need if results.get(l) != 'pass']
    return ('OPEN', []) if not blockers else ('CLOSED', blockers)
print("gate to tether:", gate('tether'), "| gate to HIL:", gate('HIL'))
assert gate('HIL')[0] == 'OPEN' and gate('tether') == ('CLOSED', ['ground'])

Common engineering mistakes

Failure modes

Debugging approach

1. For every flight defect, ask which level should have caught it and add the test there.
2. Keep a defect log with the level of discovery and the level it should have been found.
3. Check HIL timing with the full flight load.
4. Review the gate results before each move up.

Practical exercises

1. Fill the defect-by-level matrix for your project with probabilities and justify two entries.
2. Define the entry criteria list for the first free flight of your navigation system.
3. Design five SIL fault-injection scenarios for GNSS loss and VO failure.
4. Compute the cumulative detection before flight for a defect class with detection probabilities 0.7, 0.5, 0.4 at SIL, HIL and ground.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• NASA Systems Engineering Handbook (NASA/SP-2007-6105), product verification
• PX4: Simulation
• PX4: Test and continuous integration

What’s next