M12-T6 — A GNC Design Review

Learning objectives

After this lesson, you should be able to:

• Trace each requirement of a GNC design to its evidence and assign a status of met, not met, not demonstrated or open.
• Check that the evidence in different documents uses the same assumptions, and recompute it when it does not.
• Rank findings by the failing probability they remove, and recompute the design with the proposed actions.
• Write a disposition with actions, owners and evidence required to close each.

The intuition

A design review is the last place where an error can be found cheaply. The reviewer has not built the system and reads the documents the way a user of the evidence would. The structure of a review is the structure of the previous lessons, turned into questions. Requirements and their management are covered in UAV M8-T1: System requirements engineering and the safety argument that a review feeds in Navigation M10-T2: Engineering Trade-offs and Safety Cases; this lesson shows the quantitative part for the GNC loop.

Six questions

Is every requirement traced? For each requirement the review needs one number from one analysis or test, with the version of the design it applies to. Hopper's has four: land within 3 m in 99% of flights, keep stability margins, close the loops with the delay the software has, and survive the faults named in the hazard list.

Is the budget complete? The error budget of M12-T1: Error Budgets and Covariance Analysis lists sources. A reviewer asks which sources are missing: the beacon fault flights, a bias in the pad survey, the wind that the loop cannot reject. A budget that meets the requirement because it left something out is the commonest failure.

Do the margins hold across the envelope? The policy of M12-T3: A Stability-Margin Policy states the envelope; the reviewer checks that the sweep covered it, and at which corner it was tight.

Is the demonstration sufficient? The campaign of M12-T2: Monte Carlo Dispersion Analysis gives a bound; the reviewer checks the number of runs against the allowed failures, and that the dispersions are the ones in the budget.

What is not covered? The fault architecture of M12-T4: Modes, Redundancy and Fault Management in GNC lists faults it detects and ones it does not; the reviewer asks for a layer for each of the second kind.

Does the code do what the model did? The timing and number-format checks of M12-T5: From Model to Flight Software are the bridge: the delay in the code must be the delay in the margin sweep.

The finding that no single document shows

Each of the five documents is internally correct. Read together, they disagree: the budget gives a spread of 1.10 m per axis (97.4% success once the beacon-fault flights are included) and the campaign was run with a spread of 0.95 m (99.2%). Neither is wrong about its own model; the campaign would have been presented as showing near-compliance, and it was run on an optimistic design. Only a reviewer who reads both sees it.

A review compares documents with each other. The strongest findings are inconsistencies between pieces of evidence that were each produced correctly.

Definitions

Traceability

The link from a requirement to the analysis or test that shows it is met, with the version of the design to which that evidence applies.

Finding

A statement of a gap, with the evidence for it, a size, and the requirement it affects.

Disposition

The reviewer's decision: accept, accept with actions, or reject.

Action

A change or a piece of evidence, with an owner and a closing criterion.

Open risk

A hazard that nothing in the current design covers.

The success probability that the review recomputes combines the Gaussian miss of the budget with the flights that suffer a beacon fault:

The effect of a proposed action is found by recomputing the budget with the changed term and the success probability from the formula above, and the cost of a demonstration follows from the allowed failures:

Derivation: which action removes the most failure

The ranking of actions is a short calculation on the variance, with the beacon-fault flights set aside.

1. $$1 - P_{\mathrm{in}}(\sigma, 0) = \exp\!\left(-\frac{R^2}{2\sigma^2}\right)$$

   For a centred circular Gaussian the probability of missing by more than R is the exponential from module 1.

2. $$\frac{d}{d\sigma^2}\left[1 - P_{\mathrm{in}}\right] = \frac{R^2}{2\sigma^4}\,e^{-R^2/(2\sigma^2)}$$

   The slope of the failure probability against the variance: positive, so any reduction of a variance reduces the failure probability.

3. $$\Delta(1 - P_{\mathrm{in}}) \approx \frac{R^2}{2\sigma^4}\,e^{-R^2/(2\sigma^2)}\,\Delta(\sigma^2)$$

   To first order the failure removed by an action is its variance reduction times this slope, which is the same for every action. The best action is the one with the largest reduction of variance.

4. $$\Delta(\sigma^2) = \sigma_i^2 - \sigma_i'^2$$

   For the position fix, 0.75 m to 0.45 m: 0.5625 − 0.2025 = 0.360 m². For the gust bias, 0.512 m to 0.057 m: 0.262 − 0.003 = 0.259 m². The position fix removes more variance and so more failures, as in the worked example (6.4 against 10.2 failures per 1000).

The first-order argument fails for the flights with a fault: their failure probability depends on the offset $d$ against the radius, not on the spread. After the two changes of the example the fault flights (0.4% of all) account for 72% of what still fails, and no change to the spread reduces it.

The best action reduces the largest variance. Once the Gaussian part is small, the failures that remain belong to the faults, and the next action must be a fault-tolerant design, not a smaller spread.

A worked example

Review of the delivered Hopper design (the budget of M12-T1: Error Budgets and Covariance Analysis, the campaign of M12-T2: Monte Carlo Dispersion Analysis, the margin sweep of M12-T3: A Stability-Margin Policy, the fault architecture of M12-T4: Modes, Redundancy and Fault Management in GNC and the timing of M12-T5: From Model to Flight Software).

1. R1 landing accuracy. The budget gives a spread of 1.099 m per axis against 0.989 m allowed. With the 0.4% of flights that suffer a beacon fault (a 2.5 m offset) the success probability is 97.44% against 99%: not met.
2. R2 margins. 48.9° nominal with the 19 ms of delay, against 45°: met. At the corner with a 25 rad/s servo and 60 ms the margin is 29.8° against 30°: not met at the worst corner.
3. R3 and R7. The 19 ms of latency is within the 30 ms allowed for a 45° margin; numbers are in a local frame in double precision with a tick clock: both met.
4. R4 demonstration. A campaign of 10 000 runs demonstrates the requirement only with at most 83 failures; the design is expected to have 256: not demonstrated, and no number of runs would demonstrate it.
5. R5 consistency. The campaign was run with 0.95 m per axis (success 99.17%) while the budget says 1.099 m (97.44%): the evidence disagrees with itself, and the campaign result cannot be used.
6. R6 open risk. The analytic check does not see an altimeter drifting below 0.2 m/s, and there is no second dissimilar altimeter.
7. Ranking. A position fix of 0.45 m instead of 0.75 m: spread 0.921 m, success 99.36%. An integral state that removes the gust bias: 0.974 m, 98.98%. Both: 0.768 m, 99.83%, with 1.7 failures per 1000 flights, of which 1.2 (72%) belong to the beacon-fault flights.
8. Disposition. Accept with actions: a position fix of 0.45 m or better; an integral state in the lateral loop; a servo of at least 40 rad/s (worst margin 34.6°); a second dissimilar altimeter and a beacon vote for the fault flights; and a rerun of the campaign with the budget's spread, at least 800 flights for the revised design.

import math
import gnckit as g

R, P_REQ, CONF = 3.0, 0.99, 0.95
GR, LAM, B0 = 9.81, 0.7, 19.62
KP, KD = 0.8405, 0.2854
K = [0.150, 0.272, 1.658, 0.386]

# ---- the evidence, recomputed from the design (the same models as the previous lessons)
A = [[0, 1, 0, 0], [0, 0, GR, 0], [0, 0, 0, 1], [0, 0, LAM ** 2, 0]]
Bu = [0.0, GR, 0.0, B0]
Acl = [[A[i][j] - Bu[i] * K[j] for j in range(4)] for i in range(4)]
def offset(rhs): return abs(g.mat_solve(Acl, rhs)[0])
s_pos = 1.0
s_vel = offset([-Bu[r] * K[1] for r in range(4)]); s_th = offset([-Bu[r] * K[2] for r in range(4)]); s_q = offset([-Bu[r] * K[3] for r in range(4)])
s_wind = offset([0, -1.0, 0, 0])
Q = [[0.0] * 4 for _ in range(4)]; Q[1][1] = 0.30
sig_turb = math.sqrt(g.lyap(Acl, Q)[0][0])

def budget(sig_pos, wind_bias, sig_target=0.45):
    parts = {'position fix': sig_pos, 'velocity': s_vel * 0.12, 'pitch angle': s_th * 0.018, 'pitch rate': s_q * 0.006,
             'target point': sig_target, 'gust bias': s_wind * wind_bias, 'turbulence': sig_turb}
    return parts, math.sqrt(sum(v * v for v in parts.values()))

def inside(sig, cx, N=300):
    h = 2 * R / N; tot = 0.0
    for i in range(N):
        x = -R + (i + 0.5) * h
        for j in range(N):
            y = -R + (j + 0.5) * h
            if x * x + y * y <= R * R:
                tot += math.exp(-((x - cx) ** 2 + y * y) / (2 * sig * sig)) * h * h / (2 * math.pi * sig * sig)
    return tot

P_FAULT, OFFSET = 0.004, 2.5
def p_success(sig): return (1 - P_FAULT) * inside(sig, 0.0) + P_FAULT * inside(sig, OFFSET)

def binom_cdf(k, n, p):
    s = 0.0
    for i in range(k + 1):
        s += math.exp(math.lgamma(n + 1) - math.lgamma(i + 1) - math.lgamma(n - i + 1) + i * math.log(p) + (n - i) * math.log1p(-p))
    return min(1.0, s)
def k_allowed(n):
    k = -1
    while binom_cdf(k + 1, n, 1 - P_REQ) <= 1 - CONF: k += 1
    return k
def runs_for_probability(p_true, target=0.9):
    for n in range(100, 200001, 100):
        if binom_cdf(k_allowed(n), n, 1 - p_true) >= target: return n
    return None

# ---- the margin evidence: phase margin of the pitch loop from the closed form
def pm(delay, servo, b=B0):
    lo, hi = 0.5, 60.0
    mag = lambda w: math.hypot(KP, KD * w) * b / (w * w + LAM ** 2) * servo / math.hypot(w, servo)
    for _ in range(80):
        mid = (lo + hi) / 2
        if mag(mid) > 1: lo = mid
        else: hi = mid
    wc = (lo + hi) / 2
    return math.degrees(math.atan(KD * wc / KP) - math.atan(wc / servo) - wc * delay)

parts, sig = budget(0.75, 0.45)
p_before = p_success(sig)
print("REVIEW OF THE HOPPER DESIGN, as delivered")
print(f"  R1 landing accuracy: per-axis spread {sig:.3f} m against {R / math.sqrt(-2 * math.log(1 - P_REQ)):.3f} m allowed; with the beacon-fault flights the success probability is {p_before:.4f} against {P_REQ}  -> NOT MET")
pm_nom = pm(0.019, 40.0); pm_worst = pm(0.060, 25.0)
print(f"  R2 stability margins: nominal phase margin {pm_nom:.1f} deg against 45; worst corner (servo 25 rad/s, 60 ms) {pm_worst:.1f} deg against 30  -> {'MET' if pm_worst >= 30 else 'NOT MET at the worst corner'}")
print(f"  R3 latency: pitch loop delay 19 ms against the 30 ms allowed for a 45 degree margin  -> MET")
n_have = 10000
print(f"  R4 demonstration: {n_have} runs demonstrate the requirement only if at most {k_allowed(n_have)} fail; the design's expected failures are {n_have * (1 - p_before):.0f}  -> NOT DEMONSTRATED")
p_camp = p_success(0.95)
print(f"  R5 consistency: the dispersion campaign assumed a spread of 0.95 m per axis (success {p_camp:.4f}), the budget says {sig:.3f} m (success {p_before:.4f}); the evidence disagrees with itself  -> FINDING")
print("  R6 altimeter drift: not detected by the analytic check below 0.2 m/s, no second dissimilar altimeter  -> OPEN RISK")
print("  R7 number formats: local frame in double precision, tick counter  -> MET")
print()
print("the findings ranked by the share of the failing probability they could remove:")
cands = [('better position fix, 0.75 m -> 0.45 m', budget(0.45, 0.45)[1]), ('integral state removes the gust bias (0.45 -> 0.05 m/s2)', budget(0.75, 0.05)[1]),
         ('both', budget(0.45, 0.05)[1])]
for name, s_new in cands:
    p_new = p_success(s_new)
    print(f"  {name:55s}: spread {s_new:.3f} m, success probability {p_new:.4f}, failures per 1000 flights {1000 * (1 - p_new):.1f}")
parts2, sig2 = budget(0.45, 0.05)
p_after = p_success(sig2)
fault_loss = P_FAULT * (1 - inside(sig2, OFFSET))
print(f"after both changes, the share of the failures that comes from the 0.4% beacon-fault flights: {100 * fault_loss / (1 - p_after):.0f}% ({1000 * fault_loss:.1f} of {1000 * (1 - p_after):.1f} failures per 1000)")
print(f"servo requirement of at least 40 rad/s at 60 ms: worst phase margin {pm(0.060, 40.0):.1f} deg against 30  -> MET")
print()
n_new = runs_for_probability(p_after)
print(f"runs needed to demonstrate the requirement with probability 0.9 for the revised design: {n_new}; for the delivered design the true probability {p_before:.4f} is below the requirement, so no number of runs will demonstrate it")
print(f"review disposition: ACCEPT WITH ACTIONS: 1) position fix 0.45 m or better, 2) integral state in the lateral loop, 3) servo of at least 40 rad/s, 4) a fault-tolerant landing (second altimeter and a beacon vote), 5) a campaign of at least {n_new} dispersed flights")

REVIEW OF THE HOPPER DESIGN, as delivered
  R1 landing accuracy: per-axis spread 1.099 m against 0.989 m allowed; with the beacon-fault flights the success probability is 0.9744 against 0.99  -> NOT MET
  R2 stability margins: nominal phase margin 48.9 deg against 45; worst corner (servo 25 rad/s, 60 ms) 29.8 deg against 30  -> NOT MET at the worst corner
  R3 latency: pitch loop delay 19 ms against the 30 ms allowed for a 45 degree margin  -> MET
  R4 demonstration: 10000 runs demonstrate the requirement only if at most 83 fail; the design's expected failures are 256  -> NOT DEMONSTRATED
  R5 consistency: the dispersion campaign assumed a spread of 0.95 m per axis (success 0.9917), the budget says 1.099 m (success 0.9744); the evidence disagrees with itself  -> FINDING
  R6 altimeter drift: not detected by the analytic check below 0.2 m/s, no second dissimilar altimeter  -> OPEN RISK
  R7 number formats: local frame in double precision, tick counter  -> MET

the findings ranked by the share of the failing probability they could remove:
  better position fix, 0.75 m -> 0.45 m                  : spread 0.921 m, success probability 0.9936, failures per 1000 flights 6.4
  integral state removes the gust bias (0.45 -> 0.05 m/s2): spread 0.974 m, success probability 0.9898, failures per 1000 flights 10.2
  both                                                   : spread 0.768 m, success probability 0.9983, failures per 1000 flights 1.7
after both changes, the share of the failures that comes from the 0.4% beacon-fault flights: 72% (1.2 of 1.7 failures per 1000)
servo requirement of at least 40 rad/s at 60 ms: worst phase margin 34.6 deg against 30  -> MET

runs needed to demonstrate the requirement with probability 0.9 for the revised design: 800; for the delivered design the true probability 0.9744 is below the requirement, so no number of runs will demonstrate it
review disposition: ACCEPT WITH ACTIONS: 1) position fix 0.45 m or better, 2) integral state in the lateral loop, 3) servo of at least 40 rad/s, 4) a fault-tolerant landing (second altimeter and a beacon vote), 5) a campaign of at least 800 dispersed flights

In robotics

The same procedure fits a review of any GNC design.

• A review package. The evidence matrix, the list of findings with sizes, and the disposition are the output. They are written so that someone who did not attend can follow what was found and what is required to close it.
• Closing actions. Each action carries a closing criterion that is a number from an analysis or a test: "success probability at least 99.5% in the revised budget and a campaign of 800 flights with at most 4 failures in 1000". Actions that read "improve navigation" cannot be closed.
• Configuration. The evidence applies to a version of the design. A change after the review (a new servo, a new gain schedule) reopens the items that depend on it: the margin sweep and the campaign, not the number formats.
• Independent reviewers. The reviewer should not have built the part under review. The inconsistency between the campaign and the budget is the kind of finding that the builders of each piece will not see.
• After the review. The accepted design goes to the test stages of UAV M8-T4: SIL, HIL and ground testing, and the plan for the flights themselves is in UAV M8-T5: Flight-test planning.

The two labs of this module put the analysis into practice: the first builds the budget and campaign and states the confidence, the capstone asks the learner to write the navigation, guidance and control of Hopper and pass the acceptance campaign.

Common misconceptions

Key takeaways

1. A review traces each requirement to one number from one analysis, with a status of met, not met, not demonstrated or open.
2. The strongest findings are inconsistencies between documents: here a campaign at 0.95 m against a budget of 1.10 m.
3. Findings are ranked by the failure each action removes; the position fix removes 6.4 failures per 1000 against 10.2 for the integral state, and both reach 1.7.
4. After the Gaussian terms are small, the remaining failures belong to faults and need a fault-tolerant design.
5. The disposition lists actions with closing numbers: a position fix of 0.45 m, an integral state, a servo of 40 rad/s or more, a second altimeter and 800 or more flights.
6. The result of this review is accept with actions, not accept.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• NASA, NASA Systems Engineering Handbook, NASA/SP-2016-6105 Rev 2, the chapters on technical reviews. Free; US government work.
• European Cooperation for Space Standardization, ECSS-E-ST-60-30C: Satellite attitude and orbit control system (AOCS) requirements, 2013, for an example of requirements, error budgets and verification at this level. Free after registration.
• D. Wiegmann and S. Shappell, A Human Error Approach to Aviation Accident Analysis, Ashgate, 2003, for why reviews of single pieces miss failures between them. Paid.
• MIT 16.851 Satellite Engineering (OpenCourseWare) , for system-level budgets and reviews. Free; licence CC BY-NC-SA 4.0.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Related content

Nothing below is required to finish this lesson.

• Before
  M12-T5: From Model to Flight Software

  Assign rates and a latency budget to the GNC functions, list what changes between a model and fixed-step flight code, and reconstruct a flight from its data.

Other courses on this site treat neighbouring topics from their own angle:

• Probability M1-T3: Simulating Randomness: Sampling and Monte Carlo
• Probability M4-T4: Propagating Uncertainty Through Transformations
• Linear Algebra M9-T4: Covariance Propagation
• UAV M8-T9: Fault detection, isolation and recovery
• UAV M8-T7: FMEA
• UAV M8-T3: Verification and validation
• UAV M8-T4: SIL, HIL and ground testing
• UAV M6-T14: Software-in-the-Loop simulation
• UAV M7-T7: Jitter and deadline monitoring
• UAV M8-T1: System requirements engineering
• UAV M8-T5: Flight-test planning
• Navigation M10-T2: Engineering Trade-offs and Safety Cases
• Robot Learning M11-T2: Seeds, Variance and Statistics

This course leaves the following to the pages that already teach them:

• Requirements engineering, FMEA and fault-tree analysis as methods
• SIL and HIL test benches, real-time scheduling and time synchronization
• Flight-test planning and incident investigation