M10-T3 — System Identification for Transfer

Learning objectives

After this lesson, you should be able to:

• Fit gain and drag from a log of velocities and commands with RLCore.ridgeFit, and extend the regression with a command lag to identify a delay.
• Show with a lag scan why a fit that does not model the delay gives wrong parameters, and what its one-step residual looks like.
• Compare training in the calibrated world, training over ranges, and both, over several seeds and delays, with intervals, and say which differences are within noise.
• State the limits: calibration is only as good as the model structure, the log and the noise assumptions; it can go stale; and a good policy score does not prove a good model.

The intuition

Recall the model of module 7: on each axis, v_t+1 − v_t = 0.25 (gain a_t − drag v_t) + noise. Least squares recovers gain and drag from a log of (v, a, v'). The floor has one more thing, a delay of one step: the command a_t reaches the wheels at t + 1. If the regression pairs v_t+1 − v_t with a_t, it pairs a velocity change with the wrong command. When the commands change every step (random), a_t says nothing about the change that follows it, and the fit finds a gain of about zero: in my runs −0.01 on average over five data seeds. When the commands are held for four steps, a_t partly resembles a_t−1, and the fit finds a plausible but wrong robot: gain 1.17 and drag 1.74 instead of 2.2 and 3.4. Both fits have the same remedy: let the regression use a_t−k and choose the lag k with the smallest residual.

The figure shows that residual for both logs. At the true lag of 1 it is 0.0036 for both. At the other lags it is 0.03 to 0.11, 8 to 30 times larger. The residual is a diagnostic you can read without ever training a policy.

Having a good model of the floor is the first step. The second is to use it, and the interesting question is how: train in the calibrated world only, randomize around it, or ignore it and randomize widely. The rest of the lesson measures those.

The mathematics

Regression with a lag

For each axis and each time t in the log, with measured velocity v and command a, the lag-k model is

k

The assumed number of steps between command and effect: the delay of the model. k = 0 is the module-7 regression.

α, β

The two regression coefficients, found by RLCore.ridgeFit(X, Y, 0, false) on the rows [v_t, a_t−k] and targets v_t+1 − v_t (both axes stacked).

residual

The mean of e² over the log. It cannot fall below the noise in the measured velocities, so the true lag is where it is smallest.

stand-still noise

With the robot parked and a zero command, the spread of the velocity readings is the observation noise, measured directly without any model.

Choosing the lag. Fit for k = 0, 1, 2, 3, ... and take the k with the smallest residual. This picks the lag at which the logged commands best explain the logged velocity changes. It works because of one assumption: the commands are not perfectly predictable from each other, otherwise the lags are indistinguishable. A log that holds a constant command gives no information about the delay.

Not everything is identified. The process noise cannot be separated from the sensor noise by this regression: the residual contains both, and the noisy reading appears in the regressor, which biases the drag slightly (lesson M7-T2). In the experiments below I set the calibrated world’s process noise to 0 (not identified) and its observation noise to the stand-still estimate.

Three ways to use a model

A worked example

All numbers are real runs in Node. The log: 400 steps on the floor (gain 2.2, drag 3.4, delay 1, obsNoise 0.05, noise 0.03), in segments of 20 steps from random starts, commands uniform in [−1, 1] on each axis, velocities as measured (with noise). "Random" changes the command every step, "held" every 4 steps. Training: trainTheta(ranges, seed, 30), seeds 1 to 5, with the data seed equal to the training seed; evaluation: 5 seeds x 40 starts = 200 episodes on the real floor, seeds 101 to 105 (so the policy sees evaluation starts it was not trained on).

By hand: why the lag matters

Scout has v = 0.4 m/s, the command now is 1 and the previous command was −1. The model without a delay predicts v' − v = 0.25 (2.2 × 1 − 3.4 × 0.4) = 0.21. The floor, which applies the previous command, gives 0.25 (2.2 × (−1) − 3.4 × 0.4) = −0.89. The sign is wrong and the error is 1.1 m/s per step. A model with a lag of one step is exact here.

Does the fit find the lag, and the parameters?

The residual picked the lag correctly in all 5 data seeds for both kinds of log, and in 20 of 20 data seeds (held commands) for logs of 50, 100, 200, 400 and 800 steps. At the chosen lag the held-command log with 400 steps gave, over 20 data seeds, gain 2.218 ± 0.024 and drag 3.429 ± 0.040 (mean ± standard deviation); with 50 steps, 2.218 ± 0.061 and 3.443 ± 0.087; with 800, 2.213 ± 0.016 and 3.427 ± 0.024. The small upward bias of about 0.01 to 0.02 in the gain does not shrink with data (sensor noise in the regressor, as in lesson M7-T2). The stand-still readings gave an observation noise of 0.050 from 100 readings (0.0497 from 20). Beware the naive fit: the held-command log at lag 0 gives a top speed gain / drag of 1.17 / 1.74 = 0.67 m/s, close to the floor’s 0.647, so a quick check of the final speed would pass, while the response time (1 / drag) is 0.57 s against 0.29 s. The speed was learned from the held commands, the dynamics were not. Also, the same fit on a floor with a delay of 3 chose lag 3 (residual 0.0036 against 0.0304 to 0.0543 at the others), gain 2.20, drag 3.43.

Which world to train in

The floor itself, delay 1 (successes out of 200, mean steps in brackets; 95% Wilson intervals about 96% to 100% for 198 or more, 63% to 76% for the nominal-only row):

How to read it, with the honesty it needs.

• At the floor’s own delay, every strategy that is not nominal-only gets 200 of 200. Even a world with the right dynamics and no delay trained a cautious policy (clearance 0.7 to 0.9 with feedback 2), because the noise in that world made the grid prefer one. This toy cannot tell calibration from randomization at the floor.
• The delay missing matters when the delay is larger. At a delay of 3, exact dynamics without the delay gave 147 of 200 (95% interval 67% to 79%); fitting the lag gave 198 (96% to 100%). That is a real difference (the intervals do not overlap), and it is the one place where "calibrating cannot fix an unmodelled delay" shows up in the policy.
• A bad calibration got lucky. The naive log fit (wrong gain and drag, no delay) gave 193 at a delay of 3, better than exact dynamics without the delay (147), because its sluggish, low-drag world made the grid choose the cautious clearance with feedback. I would not claim anything about it other than that a wrong model can happen to push the policy the right way, and that this is not a method.
• Randomizing the delay up to 4 or calibrating it exactly differ by little. At delay 3, 190, 194, 197 and 198 of 200 (the randomized, wide-randomized, both and calibrated rows) have overlapping intervals; I cannot rank them with five seeds. The steps differ more clearly: the wide ranges cost 1.8 steps at delay 3 over the calibrated one (50.4 against 48.6), the same price as in lesson M10-T2.
• Nothing helps at delay 4. Even the calibrated world, which knows the delay, gives 149 of 200 (68% to 80%): the policy family has no member that handles it. A calibration cannot repair what the policy class cannot do, and randomizing does not either.
• Training with the delay-0 data on a log of random commands gives gain −0.01. Clamped to 0.05 to build a world, the policy (clearance 0.9, kp 0) still got 200 of 200 on the floor but took 47.8 steps, 4 more than the others: a world in which the robot barely moves is a world where the grid picks something wide.

When the floor moves

Suppose the calibration was done on the floor above, and then the floor changes (gain 1.8, drag 3.9, delay 2). Policies trained on the old calibration with a lag fit succeeded in 199 of 200, randomized ones 199, the oracle (trained on the new floor) 199, nominal-only 121. Here the cautious policy has enough headroom for the drift. With a larger drift, a calibration is stale; it must be redone (lesson M7-T2), and randomizing around the calibration (the "both" row) is the cheap insurance. I did not run a larger drift beyond the delay sweep above.

What the toy simplifies. The model structure of the fit is exactly the structure of the floor, which is the thing a real robot never offers; the policy has three numbers on a coarse grid, so most strategies end up at the same cautious member; and the log is clean, long and made by me. The calibrated process noise was set to zero, not identified.

Engineering implications and limitations

A procedure for calibration before transfer

• Log with variety and with time stamps. Commands that change (random or steps of different sizes), several speeds, resets; and time stamps for commands and for measurements so that the lag can be seen. A log of held commands only teaches the top speed.
• Scan the lag before you trust the parameters. Fit for several k and read the residual (figure). A minimum that is deep and unique means the delay is identified; a flat residual means the log cannot tell you (or the model is missing something else).
• Measure sensor noise at rest. It is a model-free number, and it helps to separate sensing from dynamics.
• Validate on a held-out log with different commands, one step and many steps ahead (lesson M7-T3), and check the residuals against the speed and the command.
• Judge the model, not only the policy. Here policies trained on a model with gain −0.01 or 1.17 scored 200 of 200 at the floor’s delay. The score said nothing about the model; the residual did.
• Calibrate, then randomize around it. Ranges centred on the identified values with widths from the estimates’ uncertainty (here ±15% for gain and drag, k±1 for the delay) used what was measured and kept the headroom; this matches the combination that Tan et al. describe in words. It costs steps only if the ranges are wider than the uncertainty (lesson M10-T2).
• Re-identify when the robot changes: payload, wear, floor, software. A calibration has a date.
• Report seeds and intervals (lesson M11-T2). Differences of 190 against 198 out of 200 on five seeds are not conclusions.

Choosing

Common misconceptions

Engineering takeaways

1. Calibrate with a regression on logged velocities and commands (RLCore.ridgeFit); to identify a delay, scan the command lag and take the minimum residual: 0.0036 at the true lag of 1 against 0.03 to 0.11 elsewhere.
2. Ignoring the delay gives wrong parameters: gain −0.01 for random commands, 1.17 and drag 1.74 for held ones (top speed nearly right at 0.67 m/s, response time wrong).
3. In this toy the policy tolerated the wrong models at the floor’s own delay (200 of 200 for every non-nominal strategy), so the policy score is a poor check of the model.
4. At a delay of 3, exact dynamics without the delay gave 147 of 200, fitting the lag 198, randomizing 190 to 194, calibrating plus modest randomization 197; the differences among the last four are inside the noise of five seeds.
5. Calibrate when you can log the robot and trust the structure, randomize around the calibration for headroom, and re-identify when the robot changes; the whole study is a toy with a coarse policy grid and no real data.

References and further reading

Ljung is the standard text on identification. Tan et al. and Hwangbo et al. combine identification of actuator models with simulation training on real quadrupeds; I describe their ideas in words and do not reproduce them. My numbers are from the toy.

• L. Ljung, “System Identification: Theory for the User”, 2nd edition, Prentice Hall, 1999.
• J. Tan et al., “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots”, Robotics: Science and Systems (RSS), 2018.
• J. Hwangbo et al., “Learning agile and dynamic motor skills for legged robots”, Science Robotics 4(26), 2019.
• X. B. Peng, M. Andrychowicz, W. Zaremba and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization”, IEEE International Conference on Robotics and Automation (ICRA), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Lesson M10-T4 moves from fixing the gap to measuring a policy’s sensitivity to it: stress tests that change friction, noise, delay and actuator response one at a time and report a robustness curve with intervals, the thing lab M10-L2 asks you to write.

Related content

Nothing below is required to finish this lesson.

• Before
  M10-T2: Domain Randomization

  Choose which parameters to randomize and over what ranges, and say when randomization helps and when it hurts.

• Lab
  M10-L4: Plan a Staged Rollout

  Choose the stages, limits and rollback triggers for a first deployment and see which risks each one removes.