M5-T5 — Adaptive Control and Its Risks

Learning objectives

After this lesson, you should be able to:

• Derive the gradient adaptation law of a model-reference controller from a Lyapunov function.
• State what the proof guarantees about the tracking error and the gains, and what it does not.
• Demonstrate parameter drift with a noisy measurement and no excitation, and the effect of leakage.
• Explain how a delay and a large adaptation gain destroy the loop.

The intuition

The last lesson's hybrid learning and control counterpart learns a model offline. Adaptive control learns a controller online, from the error, in closed loop, and stability depends on how it is done.

The idea

A reference model says how the vehicle should behave: for the scalar example, $\dot{x}_m = -a_m x_m + b_m r$, a first-order response to the command $r$. The plant is $\dot{x} = a x + b u$ with $a$ unknown (and positive, so the plant alone is unstable). A control $u = k_x x + k_r r$ makes the plant match the model for exactly one pair of gains, $k_x^* = -(a + a_m)/b$ and $k_r^* = b_m/b$. The adaptive law adjusts $k_x$ and $k_r$ in the direction that shrinks the difference between plant and model.

Why it is stable

The law is chosen so that a quantity made of the tracking error and the gain errors, $V = e^2/2 + (b/2\gamma)(\tilde{k}_x^2 + \tilde{k}_r^2)$, never grows. It is a Lyapunov function in the sense of M5-T1: Lyapunov Stability. It proves that the error is bounded and tends to zero. It does not prove that the gains reach their ideal values: the error can vanish with the gains at the wrong place.

What the proof leaves out

Three things appear in simulation and not in the proof. First, with no command to excite the loop and a little noise in the measurement, the gain drifts away from its value, because the update law integrates a signal that the error does not constrain. Second, the standard cure, leakage, bounds the drift but biases the gain. Third, a delay in the input that the proof ignores turns a large adaptation gain into instability: with a 0.1 s delay the adaptation gain of 2 is already marginal, and 5 diverges. The loop has no margin to give away, and the margin shrinks with the learning rate.

Adaptive control gets tracking from learning, and the price is a time-varying nonlinear loop that has no fixed margin. The hard part is not the law but what to do when the assumptions of the proof fail.

Definitions

Reference model

A stable linear system $\dot{x}_m = -a_m x_m + b_m r$ whose response is the one wanted from the closed loop.

Tracking error $e$

$e = x - x_m$: the plant against the model.

Parameter error $\tilde{k}$

$\tilde{k} = k - k^*$: the gap between the current gains and the ideal gains. It is not measured.

Persistent excitation

A command rich enough that the gains are pinned down by the data: for a scalar gain, a signal that does not vanish.

Leakage (σ-modification)

A term $-\sigma k$ added to the update so that the gain is pulled toward zero when the error gives no information.

Derivation: the adaptive law from a Lyapunov function

The scalar plant, the reference model and the control above.

1. $$\dot{x} = a x + b(k_x x + k_r r) = (a + b k_x)x + b k_r r$$

   The closed loop for given gains.

2. $$a + b k_x^* = -a_m, \qquad b k_r^* = b_m$$

   The ideal gains make the plant equal to the model.

3. $$\dot{e} = -a_m e + b\tilde{k}_x\,x + b\tilde{k}_r\,r$$

   Subtract the model from the plant and substitute the gain errors $\tilde{k} = k - k^*$: the error obeys a stable equation driven by the gain errors times signals.

4. $$\dot{V} = e\dot{e} + \dfrac{|b|}{\gamma}\bigl(\tilde{k}_x\dot{k}_x + \tilde{k}_r\dot{k}_r\bigr)$$

   Differentiate V ($\dot{\tilde{k}} = \dot{k}$ since the ideal gains are constant).

5. $$\dot{V} = -a_m e^2 + b\tilde{k}_x\,x e + b\tilde{k}_r\,r e + \dfrac{|b|}{\gamma}\bigl(\tilde{k}_x\dot{k}_x + \tilde{k}_r\dot{k}_r\bigr)$$

   Substitute $\dot{e}$.

6. $$\dot{k}_x = -\gamma\,\mathrm{sgn}(b)\,x e, \quad \dot{k}_r = -\gamma\,\mathrm{sgn}(b)\,r e \;\Rightarrow\; \dot{V} = -a_m e^2$$

   Choose the update so that the cross terms cancel exactly. That choice is the adaptive law.

What the conclusion is

1. $$V \ge 0,\ \dot{V} \le 0 \;\Rightarrow\; e,\ \tilde{k}_x,\ \tilde{k}_r \text{ bounded}$$

   V cannot grow, so each of its terms stays below its initial value.

2. $$\int_0^\infty e^2\,dt \le V(0)/a_m \;\Rightarrow\; e \to 0$$

   The integral of the squared error is finite and $\dot{e}$ is bounded, so by Barbalat's lemma the error tends to zero.

3. $$\tilde{k} \to 0 \text{ only if the command is persistently exciting}$$

   The theorem constrains the error, not the gains. With a constant zero command the error can vanish with the gains anywhere in the bounded set.

The adaptive law is exactly what makes the cross terms of V̇ cancel. The proof guarantees a bounded loop with vanishing error and says nothing about the gains, about noise, or about delay: those are the failures of practice.

A worked example

The scalar plant $a = 0.5,\ b = 2$ with reference $a_m = 2,\ b_m = 2$. The ideal gains are $k_x^* = -1.25,\ k_r^* = 1$.

1. Learning. With $\gamma = 2$ and a square-wave command that reverses every 5 s the error has an rms of 0.323 in the first 5 s and 0.0008 in the last 5 s of 60 s; the gains reach $k_x = -1.249,\ k_r = 0.999$.
2. No excitation, with noise. With a zero command and a measurement noise of standard deviation 0.05, the gain $k_x$ is −0.743 after 100 s and −1.745 after 300 s, wandering across and beyond the ideal −1.250: nothing in the error pins it.
3. Leakage 0.5. The gain stays at −0.248 after 100 s and −0.253 after 300 s: bounded, but at the wrong value, a fifth of the way to the ideal. Leakage trades drift for bias.
4. Delay 0.1 s. With the same square wave and $\gamma = 0.5$ the largest error in the last 10 s is 0.366; with $\gamma = 2$ it is 0.504; with $\gamma = 5, 10, 30$ the loop diverges.
5. Reading. The proof holds in all four runs for the delay-free noise-free model and fails in the last three for reasons it does not mention. A larger learning rate shortens the learning and removes the margin.

import math
import gnckit as g

# plant x' = a x + b u (a unknown), reference model xm' = -am xm + bm r. Hopper's pitch-rate loop, simplified to one state.
a, b = 0.5, 2.0
am, bm = 2.0, 2.0
kx_star = -(a + am) / b
kr_star = bm / b
print(f'ideal gains: kx* = {kx_star:.3f}, kr* = {kr_star:.3f}')

def lcg(seed):
    s = seed
    while True:
        s = (1103515245 * s + 12345) % 2147483648
        yield (s / 2147483648.0 - 0.5) * 3.4641016      # uniform with unit variance

def run(gamma, rfun, noise=0.0, leak=0.0, delay=0.0, T=60.0, dt=0.001):
    x = xm = 0.0
    kx = kr = 0.0
    rnd = lcg(12345)
    buf = [0.0] * (int(delay / dt) + 1)
    hist = []
    for k in range(int(T / dt)):
        t = k * dt
        r = rfun(t)
        xmeas = x + noise * next(rnd)
        e = xmeas - xm
        u = kx * xmeas + kr * r
        buf.append(u); ud = buf.pop(0)
        dx = a * x + b * ud
        dxm = -am * xm + bm * r
        dkx = -gamma * xmeas * e - leak * kx
        dkr = -gamma * r * e - leak * kr
        x += dt * dx; xm += dt * dxm; kx += dt * dkx; kr += dt * dkr
        hist.append((t, x, xm, kx, kr, e))
    return hist

square = lambda t: 1.0 if int(t / 5.0) % 2 == 0 else -1.0
h = run(2.0, square)
rms = lambda seg: math.sqrt(sum(s[5] ** 2 for s in seg) / len(seg))
print(f'square-wave command, gamma = 2: error rms {rms(h[:5000]):.3f} (first 5 s), {rms(h[-5000:]):.4f} (last 5 s); kx = {h[-1][3]:.3f}, kr = {h[-1][4]:.3f}')

zero = lambda t: 0.0
print('no command (nothing excites the loop), noise on the measurement of x with std 0.05:')
for leak in (0.0, 0.5):
    h = run(2.0, zero, noise=0.05, leak=leak, T=300.0, dt=0.002)
    print(f'  leakage {leak}: kx after 100 s = {h[50000][3]:+.3f}, after 300 s = {h[-1][3]:+.3f}  (ideal {kx_star:+.3f})')

print('input delay 0.1 s, square-wave command, larger adaptation gain:')
for gamma in (0.5, 2.0, 5.0, 10.0, 30.0):
    h = run(gamma, square, delay=0.1, T=40.0)
    mx = max(abs(s[5]) for s in h[-10000:])
    print(f'  gamma = {gamma:5.1f}: largest tracking error in the last 10 s = ' + (f'{mx:.3f}' if mx < 1e3 and mx == mx else 'diverged'))

ideal gains: kx* = -1.250, kr* = 1.000
square-wave command, gamma = 2: error rms 0.323 (first 5 s), 0.0008 (last 5 s); kx = -1.249, kr = 0.999
no command (nothing excites the loop), noise on the measurement of x with std 0.05:
  leakage 0.0: kx after 100 s = -0.743, after 300 s = -1.745  (ideal -1.250)
  leakage 0.5: kx after 100 s = -0.248, after 300 s = -0.253  (ideal -1.250)
input delay 0.1 s, square-wave command, larger adaptation gain:
  gamma =   0.5: largest tracking error in the last 10 s = 0.366
  gamma =   2.0: largest tracking error in the last 10 s = 0.504
  gamma =   5.0: largest tracking error in the last 10 s = diverged
  gamma =  10.0: largest tracking error in the last 10 s = diverged
  gamma =  30.0: largest tracking error in the last 10 s = diverged

In robotics

Adaptive control has flown, and the experience shaped the way it is used.

• Where it is used. Adaptive elements have been added to flight controllers (the X-15 programme of the 1960s, later experiments with the NASA Intelligent Flight Control System and the F-15 and others) to recover from failures and damage. The standard modern use is a fast adaptive element on top of a fixed baseline controller, so a failure of the adaptation leaves a safe loop.
• Modification terms. Leakage, a dead zone, parameter projection and the use of both the tracking error and a prediction error are the standard guards against drift and noise, each with its cost in bias or in accuracy.
• Delay and learning rate. The adaptation gain has to be small enough that the loop with the actuator delay and lag keeps its margin. Methods that filter the control signal (the L1 architecture) decouple the learning rate from the loop bandwidth, at the price of the filter.
• Certification. The loop is nonlinear and time-varying, its gains depend on history, and its worst case is hard to bound. Gain and phase margins have no direct meaning, so certification relies on bounds from analysis, extensive simulation and monitors.
• The alternative. A well-chosen schedule (M5-T2: Gain Scheduling) or an INDI loop (M5-T3: Dynamic Inversion and Its Incremental Form) covers many of the cases adaptation is proposed for, with margins that can be measured.

The robustness tools of M5-T7: Robustness Beyond Gain and Phase Margin are what a designer uses to put a number on the loss of margin seen above.

Common misconceptions

Key takeaways

1. Model-reference adaptive control adjusts the gains so that the plant follows a reference model; the ideal gains are $k_x^* = -(a + a_m)/b$ and $k_r^* = b_m/b$.
2. The law $\dot{k} = -\gamma\,\mathrm{sgn}(b)\,(\text{signal})\,e$ comes from choosing the update that cancels the cross terms in the derivative of a Lyapunov function.
3. The proof gives bounded signals and a vanishing error, and gain convergence only for a persistently exciting command.
4. Without excitation, noise makes the gain drift (−0.74 to −1.75 over 300 s in the example); leakage bounds the drift and leaves a bias.
5. A delay of 0.1 s makes adaptation gains of 5 and above diverge: the learning rate is limited by the loop, not by the proof.
6. Adaptive elements are used on top of a fixed baseline, with modifications and monitors, because they are hard to certify.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• K. J. Åström and R. M. Murray, Feedback Systems, 2nd edition, Princeton University Press, 2021 , the adaptive-control section of the nonlinear chapter. Free to read as the authors' PDF; the book is copyrighted.
• MIT 2.160 Identification, Estimation and Learning (OpenCourseWare) , which covers adaptive control with parameter estimation. Free; licence CC BY-NC-SA 4.0.
• P. A. Ioannou and J. Sun, Robust Adaptive Control, Prentice Hall, 1996, The standard treatment of drift, leakage and projection. Free to read on the first author's page (no licence statement).
• C. Rohrs, L. Valavani, M. Athans and G. Stein, "Robustness of continuous-time adaptive control algorithms in the presence of unmodeled dynamics", IEEE Transactions on Automatic Control 30(9), 1985, the paper that showed the loss of margin. Paid.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Related content

Nothing below is required to finish this lesson.

• Before
  M5-T4: Sliding-Mode Control

  Design a sliding surface and a switching law for a second-order system, and remove chattering with a boundary layer at a stated cost in accuracy.

Other courses on this site treat neighbouring topics from their own angle:

• Lab: Build a CBF Safety Filter
• Navigation M7-T2: Safety Filters: Control Barrier Functions
• UAV M5-T5: Attitude control
• UAV M5-T16: Disturbance rejection and wind compensation
• UAV M1-T6: Roll, pitch, yaw and thrust generation
• Linear Algebra M6-T7: The Moore-Penrose Pseudoinverse
• Linear Algebra M8-T9: Null-Space Control
• Robot Learning M9-T4: Hybrid Learning and Control Systems

This course leaves the following to the pages that already teach them:

• Control barrier functions and safety filters
• Geometric attitude control of a multirotor on SO(3)
• Disturbance observers for wind
• Learned controllers and residual policies