Validating controller performance

1. Engineering problem

“It feels good” does not survive a payload change. Written requirements on overshoot, rise time, settling time and steady-state error, measured on the real robot under the stated conditions, make a controller releasable.

2. When is this needed?

• After tuning, before release.
• After any hardware or parameter change.
• When comparing controllers.

3. Selection and design decisions

Standard step-response metrics: overshoot (peak above target as a percentage), rise time (time to reach 90 % of the target), settling time (time after which the response stays within a band, here 2 %), and steady-state error (final difference). Apply them to velocity steps, and to tracking tests (RMS error over a reference path).

4. Advantages and disadvantages

• Tighter requirements need better actuators and sensors.
• More conditions (loads, speeds, surfaces, battery levels) show real robustness and take time.
• Average and worst-case values should both be reported.

5. Requirements and constraints

• Requirements are written before the tests.
• Tests cover the load, speed and battery range of the ODD.
• Raw logs are saved with the software version.

6. Architecture

reference step ─▶ controller ─▶ robot ─▶ measured response ─▶ metrics ─▶ compare with requirements

7. Implementation

1. Write requirements per metric and condition.
2. Run step tests at several speeds, loads and battery levels.
3. Compute the metrics (below) for each.
4. Compare with the requirements and note the worst case.
5. Fix and repeat; keep the logs.

def metrics(t, y, target=1.0):
    peak = max(y)
    overshoot = max(0.0, (peak - target) / target * 100)
    rise = next((a for a, b in zip(t, y) if b >= 0.9 * target), None)
    settle = None
    for i in range(len(y)):
        if all(abs(v - target) <= 0.02 * target for v in y[i:]):
            settle = t[i]; break
    sse = abs(target - y[-1])
    return overshoot, rise, settle, sse
import math
t = [i * 0.05 for i in range(81)]
y = [1 - math.exp(-3 * a) * (math.cos(6 * a) + 0.5 * math.sin(6 * a)) for a in t]
o, r, s, e = metrics(t, y)
print(f"overshoot {o:.1f} %, rise time (90 %) {r:.2f} s, settling time (2 %) {s:.2f} s, steady-state error {e:.4f}")
req = {"overshoot": 10.0, "rise": 0.5, "settle": 1.5}
print("meets requirements:", o <= req["overshoot"] and r <= req["rise"] and s <= req["settle"])

Printed (checked by the site’s tests):

overshoot 20.5 %, rise time (90 %) 0.35 s, settling time (2 %) 1.25 s, steady-state error 0.0000
meets requirements: False

Reading it: a sampled step response has 20.5 % overshoot, a 0.35 s rise time, a 1.25 s settling time and no steady-state error; against the example requirements (10 %, 0.5 s, 1.5 s) the overshoot fails, so the controller does not pass. The response is synthetic and illustrative.

8. Real examples

• Velocity loop acceptance: nine step tests (three speeds by three loads) all within the written limits.

9. Failure scenarios

• Writing the requirements after seeing the results.
• Testing unloaded only.
• Averaging away a bad worst case.

10. Troubleshooting

1. If overshoot fails, lower I or P, or add anti-windup.
2. If rise time fails, check saturation and loop rate.
3. If settling is slow, check friction and the integrator.

11. Validation

• The metric example prints the lines shown (executed by the site tests).
• All conditions in the test matrix meet the requirements, with logs kept.

12. Engineering checklist

• Requirements first
• Matrix of conditions
• Metrics computed
• Worst case reported
• Logs and versions saved

13. Further references

• Step-response metrics are standard definitions in control texts; band and thresholds vary, so state yours.

Related on FixTheRobot