M8-T9: Fault detection, isolation and recovery

Learning objectives

After this lesson, you should be able to:

• Describe the FDIR chain and the role of the policy table.
• Choose recovery actions (continue, degrade, hold, return, land, terminate) for typical UAV faults.
• Check a policy for completeness, determinism and a safe default.
• Verify each row by fault injection with a recorded result.

Conceptual explanation

FDIR is the system function that detects faults, isolates them, and recovers: it keeps the vehicle safe when something fails. Recovery is not improvisation; it is a table indexed by fault (and often by current mode) whose entries are actions. Common actions, in increasing severity: continue with a flag; switch to a redundant element; degrade capability (lesson M8-T10); hold position; return along a known path; land here; stop and disarm in an emergency.

Engineering intuition

Three properties make a policy trustworthy. Complete: every fault in the FMEA has a row, and anything unlisted falls to a safe default. Deterministic: one situation, one action, with explicit priority when faults overlap (a link loss during GNSS loss is its own row, not the result of two policies fighting). Monotonic in severity: a more serious fault never results in a milder action than a less serious one in the same context. All three can be checked by a program, which is why the policy is data.

Mathematical formulation

Model faults as a set $F$, modes as $M$, actions as a totally ordered set A (severity 0 … $n$). The policy is a function $\pi$ : $F\times M\to A$. Properties to check:

Timing matters as much as the action: the time from fault onset to a completed recovery is the sum of the detection delay, the decision latency and the execution time, and must be shorter than the time the hazard takes to develop (for example the time for drift to reach the clearance, lesson M8-T11):

Architecture and data flow

Health records from monitors (lesson M3-T16) → fault set → policy lookup (with the current mode) → recovery command to the navigation manager and mission layer, and, for hard actions (land, disarm), to PX4 through vehicle commands or the configured failsafe parameters (lesson M8-T14). Every decision logs: time, fault set, mode, action, reason. These logs are also the evidence in the verification report.

Real-world UAV applications

• Specifying the failsafe behaviour for Capstone 4.
• Mapping the FMEA to executable software and to PX4 failsafe configuration.
• Writing fault-injection test plans straight from the policy table.
• Reviewing a vehicle’s behaviour for overlapping faults.

Implementation example

This runs as shown: a policy as a table, with a mechanical check of completeness against the FMEA fault list, monotonicity in severity, and the combination rule.

ACTIONS = ["CONTINUE", "SWITCH_SOURCE", "DEGRADE", "HOLD", "RETURN", "LAND", "EMERGENCY_STOP"]       # ordered by severity
sev = {a: i for i, a in enumerate(ACTIONS)}
FAULT_SEVERITY = {"imu_secondary_fail": 1, "gnss_loss": 2, "gnss_spoof": 3, "camera_fail": 2, "lidar_fail": 2,
                  "all_odometry_lost": 5, "link_loss": 4, "imu_all_fail": 6, "companion_crash": 5}

POLICY = {  # (fault, mode) -> action ; a fault missing here falls to DEFAULT[mode]
    ("imu_secondary_fail", "NOMINAL"): "SWITCH_SOURCE", ("imu_secondary_fail", "DEGRADED"): "SWITCH_SOURCE",
    ("gnss_loss", "NOMINAL"): "DEGRADE", ("gnss_loss", "DEGRADED"): "CONTINUE",
    ("gnss_spoof", "NOMINAL"): "DEGRADE", ("gnss_spoof", "DEGRADED"): "DEGRADE",
    ("camera_fail", "NOMINAL"): "SWITCH_SOURCE", ("camera_fail", "DEGRADED"): "HOLD",
    ("lidar_fail", "NOMINAL"): "SWITCH_SOURCE", ("lidar_fail", "DEGRADED"): "HOLD",
    ("all_odometry_lost", "NOMINAL"): "HOLD", ("all_odometry_lost", "DEGRADED"): "LAND",
    ("link_loss", "NOMINAL"): "RETURN", ("link_loss", "DEGRADED"): "LAND",
    ("companion_crash", "NOMINAL"): "HOLD", ("companion_crash", "DEGRADED"): "LAND",
    ("imu_all_fail", "NOMINAL"): "EMERGENCY_STOP", ("imu_all_fail", "DEGRADED"): "EMERGENCY_STOP",
}
DEFAULT = {"NOMINAL": "HOLD", "DEGRADED": "LAND"}                                                  # safe default for anything not listed
MODES = ["NOMINAL", "DEGRADED"]

def decide(faults, mode):
    acts = [POLICY.get((f, mode), DEFAULT[mode]) for f in faults]
    return max(acts, key=sev.get) if acts else "CONTINUE"                                          # simultaneous faults: the most severe action wins

# 1) completeness: every FMEA fault has an explicit row in every mode
missing = [(f, m) for f in FAULT_SEVERITY for m in MODES if (f, m) not in POLICY]
assert not missing, missing
# 2) monotonic: a more severe fault is never answered more mildly than a less severe one, in the same mode
for m in MODES:
    for f1 in FAULT_SEVERITY:
        for f2 in FAULT_SEVERITY:
            if FAULT_SEVERITY[f1] >= FAULT_SEVERITY[f2] + 2:                                         # compare faults two severity levels apart
                assert sev[POLICY[(f1, m)]] >= sev[POLICY[(f2, m)]], (m, f1, f2)
# 3) overlapping faults resolve to the more severe action, and unknown faults hit the safe default
assert decide(["gnss_loss", "link_loss"], "NOMINAL") == "RETURN"
assert decide(["camera_fail", "lidar_fail"], "DEGRADED") == "HOLD"
assert decide(["something_new"], "DEGRADED") == "LAND" and decide([], "NOMINAL") == "CONTINUE"
print("policy: %d rows, complete, monotonic, with a safe default" % len(POLICY))

Common engineering mistakes

Failure modes

Debugging approach

1. Run the completeness and monotonicity checks in CI on the policy file.
2. Inject each fault in each mode in simulation and compare the commanded action with the table.
3. Check that every decision logs fault, mode, action and reason.
4. Measure $t_{\text{detect}}+t_{\text{decide}}+t_{\text{execute}}$ for the slowest rows.

Practical exercises

1. Add the fault “barometer fail” with a policy row in both modes and re-run the checks.
2. Introduce a non-monotonic row on purpose and see the check fail.
3. Add a third mode (DEAD_RECKONING) and fill in its rows.
4. Map each action to the PX4 parameter or command that executes it in your version.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• SAE ARP4761: safety assessment process (fault trees, FMEA, common-cause analysis)
• PX4: safety configuration (failsafe actions and parameters)
• NASA Systems Engineering Handbook (fault management)

What’s next