M9-T1 — The Optimal Control Problem and the Minimum Principle

Learning objectives

After this lesson, you should be able to:

• State an optimal control problem with a cost, dynamics, boundary conditions and an input bound.
• Write the Hamiltonian and the costate equations and state the minimum principle.
• Derive the one-switch bang-bang solution of the minimum-time double integrator.
• Derive the bang-off-bang solution of the minimum-fuel problem and its fuel as a function of time.

The intuition

UAV M5-T9: Trajectory generation builds smooth polynomial trajectories by minimizing snap, a problem with no actuator limit, and dynamic programming solves optimal control by tabulating a value function. This lesson uses a third route, calculus: the necessary conditions that an optimal control must satisfy, which for a bounded input say that the control is at a limit whenever a certain function is nonzero.

Cost, dynamics, constraints

An optimal control problem has four ingredients: the dynamics $\dot{\mathbf{x}} = f(\mathbf{x}, u)$, the boundary conditions (where it starts, where it must end), the limits on the input (here $|u| \le u_{\max}$), and a cost to minimize, usually an integral of a running cost $L(\mathbf{x}, u)$. Minimum time has $L = 1$; minimum fuel has $L = |u|$; minimum energy has $L = u^2$.

The minimum principle in one idea

Imagine a price attached to each state variable, a costate, that says how much one more unit of that variable is worth to the final cost. Pontryagin's principle says that at the optimum the control, at every instant, minimizes the Hamiltonian $H = L + \mathbf{p}^{\mathsf{T}} f$, the instantaneous cost plus the value of the change in state it causes, over all allowed controls. When $H$ is linear in the input, its minimum over a bounded interval is at one end, wherever the coefficient of $u$ (the switching function) is nonzero: the control is at a limit, and it switches when the switching function changes sign.

Hopper's attitude slew

Take Hopper's pitch angle, turned from 0.2 rad to rest at zero as fast as possible with the gimbal at most 0.15 rad. The gimbal limit means an angular acceleration of at most $a_{\max} = 19.62 \times 0.15 = 2.943$ rad/s² either way. The minimum principle gives a bang-bang control with one switch, at half the total time: accelerate toward zero at full gimbal, then decelerate at full gimbal the other way. The time is $T = 2\sqrt{\theta_0/a_{\max}}$ = 0.521 s. In the phase plane the control follows a parabola until it meets the switching curve, then slides along it into the origin.

When time is not the cost

If propellant matters more than time, the cost is the integral of the thrust. The same principle gives a third phase: bang-off-bang. Push at full gimbal for a short time, coast, and brake at full gimbal. For a total time of 1.5 s the fuel is 0.275 rad/s (an angular-velocity impulse), against 0.40 for the smooth slew that minimizes energy: the optimal fuel is 0.69 of the smooth one, and tends to two thirds of it for long times.

A bounded linear input makes the Hamiltonian linear in the control, so the best control is at a limit, or off: bang-bang for time, bang-off-bang for fuel. The principle gives the structure; the switching times come from the boundary conditions.

Definitions

Optimal control problem

Minimize $J = \phi(\mathbf{x}(T)) + \int_0^T L(\mathbf{x}, u)\,dt$ over controls $u(t)$ subject to $\dot{\mathbf{x}} = f(\mathbf{x}, u)$, boundary conditions and $u \in U$.

Costate $\mathbf{p}(t)$

The sensitivity of the optimal cost to the state; it obeys $\dot{\mathbf{p}} = -\partial H/\partial\mathbf{x}$.

Hamiltonian $H$

$H(\mathbf{x}, u, \mathbf{p}) = L(\mathbf{x}, u) + \mathbf{p}^{\mathsf{T}} f(\mathbf{x}, u)$.

Switching function

The coefficient of the input in the Hamiltonian when $H$ is linear in $u$; the sign of it decides which limit the control sits at.

Bang-bang, bang-off-bang

Controls that take only extreme values (or extreme values and zero), with a finite number of switches.

Derivation: minimum time and minimum fuel for the double integrator

State $x_1 = \theta$, $x_2 = \omega$, dynamics $\dot{x}_1 = x_2$, $\dot{x}_2 = u$, with $|u| \le a_{\max}$.

1. $$H = 1 + p_1x_2 + p_2u$$

   For minimum time the running cost is 1; the Hamiltonian adds the costate times each state derivative.

2. $$\dot{p}_1 = -\partial H/\partial x_1 = 0,\qquad \dot{p}_2 = -\partial H/\partial x_2 = -p_1$$

   The first costate is constant; the second is therefore a linear function of time: $p_2(t) = c - p_1t$.

3. $$u^* = -a_{\max}\,\mathrm{sgn}\,p_2(t)$$

   The Hamiltonian is linear in u with coefficient $p_2$: its minimum over $[-a_{\max}, a_{\max}]$ is at the limit opposite in sign to $p_2$. A linear function changes sign at most once, so there is at most one switch.

4. $$\theta(T) = 0,\ \omega(T) = 0 \;\Rightarrow\; \text{switch at } T/2$$

   Starting at rest and ending at rest means the two bursts have equal length (the velocity gained must be lost), so the switch is at the half time.

5. $$\theta_0 = 2\cdot\tfrac12 a_{\max}(T/2)^2 = a_{\max}T^2/4 \;\Rightarrow\; T = 2\sqrt{\theta_0/a_{\max}}$$

   The distance covered in the two halves, each with the magnitude of acceleration $a_{\max}$, equals the starting angle.

Minimum fuel

1. $$H = |u| + p_1x_2 + p_2u$$

   The running cost is $|u|$ (the propellant flow is proportional to the control magnitude); the time T is fixed and longer than the minimum.

2. $$u^* = \begin{cases}-a_{\max} & p_2 > 1\\ 0 & |p_2| < 1\\ +a_{\max} & p_2 < -1\end{cases}$$

   Minimize $|u| + p_2u$ over the bounds: if $|p_2| < 1$ the best is zero, because any thrust costs more than the value it buys; beyond $\pm1$ the best is a limit. Since $p_2$ is linear in time, it crosses the band once: bang, off, bang.

3. $$t_1 \text{ from } a_{\max}t_1(T - t_1) = \theta_0$$

   With burst length t₁ at each end the velocity reached is $a_{\max}t_1$, coasted for $T - 2t_1$, and the distance is $a_{\max}t_1(T - t_1)$.

4. $$\text{fuel} = \int|u|\,dt = 2a_{\max}t_1 \to 2\theta_0/T\ (T \to \infty)$$

   Solve the quadratic for $t_1$ and multiply by the two bursts of magnitude $a_{\max}$. For long times it tends to $2\theta_0/T$.

The minimum principle turns an optimization over functions into a choice at each instant: minimize H over the allowed controls. With a bounded control and a linear Hamiltonian the solution sits at the limits, and the costates, linear in time for a double integrator, fix the number of switches.

A worked example

Hopper's pitch slew from 0.2 rad to rest at zero, with $a_{\max} = 2.943$ rad/s² (gimbal at its 0.15 rad limit, authority 19.62).

1. Minimum time. $T^* = 2\sqrt{0.2/2.943} = 0.5214$ s with the switch at 0.2607 s. Applying −a_max then +a_max leaves the final angle at −1×10⁻¹⁴ rad and the final rate at 9×10⁻¹⁷ rad/s.
2. A wrong switch time. Switching at 0.8, 0.9, 1.1 and 1.2 of the half time leaves a final angle of +0.088, +0.042, −0.038 and −0.072 rad and a final rate of +0.307, +0.153, −0.153 and −0.307 rad/s: the terminal state is missed in proportion to the error.
3. Brute force. Scanning every single-switch programme with free total time finds the shortest that reaches the origin at 0.519 s with the switch at 0.260 s, within the scan resolution of the exact answer.
4. Minimum fuel. For T = 0.55, 0.8, 1.5, 3 and 6 s the burst lengths are 0.187, 0.097, 0.047, 0.023 and 0.011 s, and the fuel is 1.103, 0.569, 0.275, 0.134 and 0.067 rad/s. The smooth minimum-energy slew costs 0.750, 0.400, 0.200 and 0.100 for T = 0.8, 1.5, 3 and 6 s: the bang-off-bang fuel is 76%, 69%, 67% and 67% of it.
5. Feasibility. At T = 0.55 s the smooth slew needs 3.97 rad/s², above the limit of 2.94: it is not an admissible control, which is why the bang-off-bang solution is not compared with it there.

import math

# Hopper's pitch attitude with the gimbal at its limit: theta'' = u, |u| <= amax, amax = b * 0.15 rad = 2.943 rad/s2 (the unstable term is ignored for this short slew)
b, dmax = 19.62, 0.15
amax = b * dmax
th0 = 0.2
print(f'angular acceleration limit {amax:.3f} rad/s2; slew from {th0} rad to rest at 0 rad')

# minimum time: bang-bang with one switch at half the time
T_min = 2 * math.sqrt(th0 / amax)
print(f'minimum time from the minimum principle: T = 2 sqrt(theta0 / amax) = {T_min:.4f} s, switch at {T_min / 2:.4f} s')

def simulate(u_of_t, T, n=20000):
    th, w = th0, 0.0
    dt = T / n
    for i in range(n):
        u = u_of_t(i * dt)
        th += w * dt + 0.5 * u * dt * dt
        w += u * dt
    return th, w

th, w = simulate(lambda t: -amax if t < T_min / 2 else amax, T_min)
print(f'  applying -amax then +amax: final angle {th:+.2e} rad, final rate {w:+.2e} rad/s')
for ts in (0.8, 0.9, 1.1, 1.2):
    th, w = simulate(lambda t: -amax if t < ts * T_min / 2 else amax, T_min)
    print(f'  switching at {ts:.1f} of the half time: final angle {th:+.4f} rad, final rate {w:+.4f} rad/s (the terminal state is missed)')

# a brute-force check: scan every single-switch bang-bang programme with free total time and find the shortest that reaches the origin
best = None
for i in range(200, 1500):
    T = i * 0.001
    for j in range(0, 101):
        ts = T * j / 100
        th, w = simulate(lambda t: -amax if t < ts else amax, T, n=400)
        if abs(th) < 2e-3 and abs(w) < 2e-3 and (best is None or T < best[0]):
            best = (T, ts)
print(f'brute-force scan of single-switch programmes: shortest time {best[0]:.3f} s with the switch at {best[1]:.3f} s')

# minimum fuel (cost = integral of |u|) for a longer time T: bang-off-bang
print('minimum fuel for a longer time T: accelerate for t1, coast, decelerate for t1; fuel = 2 amax t1; compare with the smooth minimum-energy slew')
print('   T (s)    t1 (s)   coast (s)   fuel int|u| dt (rad/s)   energy-optimal int|u| dt')
for T in (0.55, 0.8, 1.5, 3.0, 6.0):
    t1 = (T - math.sqrt(T * T - 4 * th0 / amax)) / 2
    fuel = 2 * amax * t1
    # minimum energy u(t) = 6 th0 (T - 2t) / T^3 (up to sign): int|u| = 6 th0 / T^3 * (T^2 / 2) = 3 th0 / T
    smooth = 3 * th0 / T
    peak = 6 * th0 / (T * T)
    note = '' if peak <= amax else f'  (the smooth slew needs {peak:.2f} rad/s2 > {amax:.2f}: not allowed)'
    print(f'  {T:5.2f}  {t1:8.4f}  {T - 2 * t1:9.4f}  {fuel:20.4f}  {smooth:18.4f}{note}')
print('for long T the bang-off-bang fuel tends to 2 th0 / T, two thirds of the 3 th0 / T of the smooth slew')

angular acceleration limit 2.943 rad/s2; slew from 0.2 rad to rest at 0 rad
minimum time from the minimum principle: T = 2 sqrt(theta0 / amax) = 0.5214 s, switch at 0.2607 s
  applying -amax then +amax: final angle -1.04e-14 rad, final rate +8.62e-17 rad/s
  switching at 0.8 of the half time: final angle +0.0880 rad, final rate +0.3067 rad/s (the terminal state is missed)
  switching at 0.9 of the half time: final angle +0.0420 rad, final rate +0.1534 rad/s (the terminal state is missed)
  switching at 1.1 of the half time: final angle -0.0380 rad, final rate -0.1534 rad/s (the terminal state is missed)
  switching at 1.2 of the half time: final angle -0.0720 rad, final rate -0.3069 rad/s (the terminal state is missed)
brute-force scan of single-switch programmes: shortest time 0.519 s with the switch at 0.260 s
minimum fuel for a longer time T: accelerate for t1, coast, decelerate for t1; fuel = 2 amax t1; compare with the smooth minimum-energy slew
   T (s)    t1 (s)   coast (s)   fuel int|u| dt (rad/s)   energy-optimal int|u| dt
   0.55    0.1874     0.1751                1.1033              1.0909  (the smooth slew needs 3.97 rad/s2 > 2.94: not allowed)
   0.80    0.0966     0.6068                0.5687              0.7500
   1.50    0.0468     1.4065                0.2752              0.4000
   3.00    0.0228     2.9543                0.1344              0.2000
   6.00    0.0113     5.9773                0.0668              0.1000
for long T the bang-off-bang fuel tends to 2 th0 / T, two thirds of the 3 th0 / T of the smooth slew

In robotics

The minimum principle explains the structure of most rocket and spacecraft trajectories.

• Rocket landing. A rocket with an engine that is either off or on at full thrust (the lab M9-L1: Find the Switch Time) has a bang-bang optimal descent: coast until a switch altitude, then burn at maximum thrust to touchdown. The switch point depends on the current height and speed and is the whole of the guidance.
• Attitude slews. A spacecraft slewed by reaction wheels or thrusters with a torque limit uses the double-integrator solution of this lesson, with a smoothing of the switch to avoid exciting flexible modes.
• Fuel against time. The fuel of the bang-off-bang solution falls as 2θ₀/T: spending longer saves propellant at a falling rate. A mission picks T from that curve and from other constraints, such as a deadline or a glide slope.
• What it does not do. It gives necessary conditions, not a solution method: for a nonlinear problem the costate boundary conditions give a two-point boundary value problem that is hard to solve, which is why the next lessons transcribe the problem to a finite-dimensional one instead.
• Chattering and real actuators. A bang-bang control asks an actuator for instant reversals. A real gimbal with a rate limit and a servo lag cannot do that, and the true minimum time is longer than 0.52 s by about the servo time constant.

The nonlinear programming approaches that replace the costate equations are the subject of M9-T2: Direct Methods: Shooting and Collocation. The least-energy law used for the explicit guidance of M8-T3: Explicit Guidance to a Terminal State is the minimum principle with cost $u^2$, whose Hamiltonian gives a control linear in the costate and so linear in time.

Common misconceptions

Key takeaways

1. An optimal control problem is a cost, dynamics, boundary conditions and an input bound; Pontryagin's principle says the control minimizes $H = L + \mathbf{p}^{\mathsf{T}}f$ at every instant.
2. For a bounded input and a Hamiltonian linear in it, the control is at a limit where the switching function is nonzero.
3. The double integrator's costate is linear in time, so the minimum-time control has at most one switch, at half the time.
4. Hopper's pitch slew of 0.2 rad with a 2.943 rad/s² limit takes 2√(θ₀/a) = 0.521 s.
5. Minimum fuel for time T is bang-off-bang with fuel 2a_max t₁, tending to 2θ₀/T: 67% of the smooth slew's 3θ₀/T for long times.
6. The principle gives the structure and necessary conditions; the switch times come from the boundary conditions.

References and further reading

Each source is marked free or paid. Free to read does not mean free to reuse: the course resources page lists every source with its licence.

• D. E. Kirk, Optimal Control Theory: An Introduction, Dover, 2004, the chapters on the minimum principle and time-optimal control. Paid.
• A. E. Bryson and Y.-C. Ho, Applied Optimal Control, Taylor and Francis, 1975. Paid.
• MIT 16.323 Principles of Optimal Control (OpenCourseWare) , the lectures on the minimum principle and bang-bang control. Free; licence CC BY-NC-SA 4.0.
• Z. Manchester, CMU 16-745 Optimal Control and Reinforcement Learning , with notes on the minimum principle. Free to read; no licence statement was found.

Knowledge check

Three questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Related content

Nothing below is required to finish this lesson.

• Lab
  M9-L1: Find the Switch Time

  Hopper falls vertically with an engine that is either off or at full thrust. Find the ignition altitude that lands with zero velocity using the least propellant, and the margin it leaves.

Other courses on this site treat neighbouring topics from their own angle:

• UAV M5-T8: Model Predictive Control
• UAV M5-T9: Trajectory generation
• UAV M5-T12: Kinodynamic planning
• Navigation M5-T6: LQR and MPC for Path Tracking
• Navigation M5-T3: Curvature, Velocity and Acceleration Limits
• Navigation M5-T2: Smoothing: Shortcuts, Splines and Gradient Methods
• Robot Learning M7-T5: Model Predictive Control with Learned Models
• Robot Learning M4-T1: Markov Decision Processes

This course leaves the following to the pages that already teach them:

• Minimum-snap polynomial trajectories and differential flatness of a quadrotor
• Path smoothing and velocity profiles for ground robots
• Sampling-based planning with a learned model (random shooting, CEM)
• Reinforcement learning