M7-T3 — Learned Dynamics and Model Error

Learning objectives

After this lesson, you should be able to:

• Measure one-step and multi-step prediction error from a rollout, and explain why they differ by an order of magnitude.
• Predict, from the dynamics, whether a model error stays bounded or explodes over the horizon, and check a prediction with the plateau speed.
• Fit a small neural network to transitions with the Module 2 tools, and show that it is accurate in its data region and worse outside it, while a physical fit is not.
• Describe how uncertainty estimates and ensembles address model error, and why distribution shift between fitting data and planner states is the central risk.

The intuition

Walking in the dark, you can estimate one step ahead very well. Do it by counting paces for a hundred steps and you end up somewhere else, because each estimate started from where the last estimate put you. A model used for planning is a dead-reckoning device, and its drift is the sum of its tiny mistakes, each fed into the next.

In the figure, the nominal simulator model (gain 3.0, drag 3.0) is asked to predict Scout on the real floor when it drives with a constant command. After one step it is off by 0.056 m, a few centimetres; after twelve, by 1.106 m. The error grew twenty times while the model never changed. The other lines say the same story from a different side: even the exact model drifts 0.070 m from a noisy world, because the noise itself accumulates; and a small network, accurate to a centimetre inside its data, is off by 0.227 m after twelve steps when asked about a command it never saw.

The mathematics

One-step error and rollout error

Let f be the real transition and f̂ the model. The one-step error is measured from a real state: ε₁(s, a) = ‖ f̂(s, a) − f(s, a) ‖. The k-step rollout error starts at a real state, feeds the model’s predictions into itself and compares with the real k-step outcome:

The recursion has two parts. A fresh error ( − A) s_j + (B̂ − B) a_j enters at every step, and the carried error  e_j is the old error pushed through the model’s own dynamics. Whether the carried error shrinks or grows is set by the eigenvalues of Â. For Scout the velocity block has the factor 1 − dt · drag (0.15 on the floor), which is below 1 in size, so velocity errors are damped; the position block integrates velocity (factor 1), so position errors keep accumulating whatever the velocity error is. If drag were negative the factor would be above 1 and the errors would grow geometrically (example 2).

Where the error comes from

Uncertainty and ensembles (description only)

A point prediction hides how much the model knows. Two kinds of uncertainty are distinguished in the model-based literature: aleatoric (noise in the world itself, which no amount of data removes) and epistemic (the model has not seen enough data here, which more data would reduce). An ensemble trains several models on resampled data or from different initialisations; where they disagree, the model is uncertain, and a planner can avoid or penalise those regions. PETS (Chua et al. 2018) combines ensembles of networks that each predict a mean and a variance with sampling-based planning. I did not run an ensemble in this lesson, so I give no numbers for it; its use for planning is the subject of lesson M7-T5.

Distribution shift

The model is fitted to transitions collected by one behaviour (random mild commands from the left of the hall). The planner then asks it about different states and actions: the ones that look good under the cost. That is the same mechanism as the drift of a cloned policy (lesson M3-T3): the data distribution of the learner’s inputs is not the one it is later used on. And it is worse than in imitation, because the planner actively searches for the inputs where the model says things are good, which are often the inputs where it is unconstrained (the model bias of lesson M7-T1).

A worked example

Numbers are from Node runs with RLCore. Start positions come from starts(seed, 200, 'all') with zero velocity. The "truth" is RLCore.dynRollout on DYN.floor (gain 2.2, drag 3.4). The commands are a constant (1, 0.5) for 12 steps or random commands in [−1, 1]. RMS means the square root of the mean squared distance over the 200 starts.

1. Nominal model against the floor: the error-by-horizon table. The nominal model (gain 3.0, drag 3.0), noiseless floor.

With a constant command, the position error at step 12 (1.106) is 4.7 times the error at step 3 (0.236), and 19.7 times the error at step 1. The velocity error, on the other hand, stops growing at 0.395. That plateau can be checked by hand: at a constant command (1, 0.5) the nominal model settles at the speeds gain / drag × a = (1.0, 0.5) m/s and the floor at 0.647 × (1, 0.5) = (0.647, 0.324) m/s, a difference of (0.353, 0.176), whose length is 0.395 m/s. Position then drifts by 0.395 × 0.25 = 0.099 m a step, and the measured slope between horizons 9 and 12 is (1.106 − 0.820) / 3 = 0.095 m a step. With random commands the signed errors partly cancel and the position error is only 0.229 at step 12: the horizon error depends on the commands used, so a model has to be tested on the commands the planner will use.

2. Error growth depends on the dynamics. A constant error in the model drag (+0.1 on the true value, gain right), random commands, from rest at (5, 3), hall walls ignored, 200 sequences. RMS position error:

All four have an identical model error of 0.1 in the drag, and a one-step position error of 0.0000 to four digits (the robot starts at rest). At step 12 the stable dynamics give 0.012 m, the unstable ones 0.60 and 1.23 m. At step 24 the last is 46.6 m. A negative drag is a hypothetical here (it makes velocity grow by itself, like a balancing robot falling); Scout’s floor is the first, forgiving, row. The lesson is that a model of an unstable system needs a much smaller one-step error for the same horizon, or a much shorter one.

3. A black-box model: a small network. Data: 2,000 transitions of random mild commands (each axis uniform in [−0.5, 0.5], so speeds up to 0.36 m/s), process noise 0.03, starting positions in x from 0.5 to 4 m, resets every 20 steps. Model: RLCore.mlpInit([6, 32, 32, 4]) with tanh, input (x, y, v_x, v_y, a_x, a_y) scaled by (5, 3, 1, 1, 1, 1), output the change in the four state variables scaled by 0.25, trained by mlpBackward and adamStep (learning rate 0.003, 400 full-batch epochs, squared error). The comparison is the physical fit from the same 2,000 rows (drag 3.407, gain 2.198, within 0.2% of the truth). "One-step" is the RMS of the 2-D velocity error against the noisy next velocity, so even a perfect model scores about 0.03 · √2 = 0.042. "12-step" is the RMS position error against the noiseless rollout. Mean over five network initialisations (standard deviation in brackets).

Inside its data, the network is as good as can be: its one-step error 0.044 is at the noise floor of 0.042, and its 12-step position error is 0.013 m. Outside, the one-step error doubles for new commands (0.090 to 0.101; for a new place alone it is 0.050) and the 12-step error is 3 to 18 times larger, up to 0.233 m (with a spread of 0.109 between initialisations) for the constant command that drives Scout to 0.65 m/s, a speed the training data never reach. The physical model, two numbers fitted to the same rows, is indifferent to where it is asked: the structure extrapolates. The network is a fair model of this world only where the data are. I used one data set size, one architecture and one training schedule, so the sizes of the gaps are properties of this setup, not general numbers.

4. The noise floor is not model error. The exact floor model, run against a world with process noise 0.05, has position errors 0.018, 0.027, 0.035, 0.050, 0.064 and 0.070 at horizons 1, 2, 3, 6, 9 and 12 (about the square root of the horizon from step 3 on). No amount of model learning removes that; it can only be handled by replanning, or by a model that predicts a distribution.

What this simulation simplifies. One state dimension per axis, a world whose true form is two parameters, Gaussian noise and an exact (noise-free) state observation. Real models fight contacts, partial observability and non-stationary dynamics. The numbers show the mechanisms, not what you will see on a robot.

Engineering implications and limitations

Evaluating a model for planning

1. Hold out whole trajectories, not random rows (adjacent rows are nearly the same transition; lesson M2-T6 on leakage). Evaluate on logs from other commands and places.
2. Report an error-by-horizon curve, not one number, for the planning horizon and the commands the planner uses. The one-step loss is the training objective and a poor guide to planning.
3. Compare with the noise floor. If the exact model’s error is 0.070 at step 12, a model at 0.075 is as good as it can be.
4. Test where the plan goes. Plan with the model, then measure the model’s error along the planned states (lab M7-L3 does this near the pillar, where a model that is accurate on average still lies).
5. Choose the horizon and the replanning interval from the curve: plan only as far as the error stays below what the task can tolerate, and replan before it grows (lesson M7-T5).

Reducing model error

• Put in structure you know (a physical form), and fit its parameters (lesson M7-T2).
• Collect data where the planner goes: after a first model, plan, run, add the new transitions and refit. This is the loop of lesson M7-T6, and it is how model-based methods fight distribution shift (cross-reference lesson M3-T3 for the same cure by DAgger).
• Train on multi-step error too, not only one step ahead, when the rollouts will be long. I did not test this here.
• Estimate uncertainty (ensembles, probabilistic models) and penalise or avoid uncertain regions; I have described this but not measured it.
• Shorten the plan and replan: error at horizon 3 is a fifth of the error at 12 in example 1.

Logging

Keep the error-by-horizon table for every model version, the data region (ranges of position, speed, command), and the noise floor, so that a later plan that leaves the region is recognised as untrustworthy. Lab M7-L1 (fit the dynamics) builds the rollout and the error-by-horizon function; lab M7-L3 (when the model lies) uses them to choose a safe horizon.

Common misconceptions

Engineering takeaways

1. One-step error is the training loss; rollout error is the planning cost. For the nominal model against the floor: 0.056 m at step 1, 1.106 m at step 12 for a constant command.
2. Error growth depends on the dynamics: damped velocity gives a plateau (0.395 m/s here) and linearly growing position error; unstable dynamics make the same parameter error explode (46.6 m at step 24).
3. Part of the multi-step error is the world’s own noise (0.070 m at step 12 for noise 0.05). Compare a model with that floor, and not with zero.
4. A network fitted to transitions is accurate inside its data (0.013 m after 12 steps) and unreliable outside (up to 0.233 m); a physical model with the right structure extrapolates (0.006 m).
5. The planner leaves the data: distribution shift (lesson M3-T3) is the central risk. Collect data where the plan goes, shorten the horizon and replan.

References and further reading

Sutton and Barto discuss model error and planning in chapter 8. PILCO accounts for model uncertainty with Gaussian processes, and PETS uses ensembles of probabilistic networks; both are cited here for the idea, not for any number in this lesson.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018, chapter 8.
• M. P. Deisenroth and C. E. Rasmussen, “PILCO: A Model-Based and Data-Efficient Approach to Policy Search”, Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.
• K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models”, Advances in Neural Information Processing Systems (NeurIPS), 2018.
• L. Ljung, System Identification: Theory for the User, 2nd edition, Prentice Hall, 1999.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lesson M7-T4 uses a model to plan, sampling action sequences (random shooting) and refining them with the cross-entropy method; lesson M7-T5 replans at every step; lesson M7-T6 adds model rollouts to learning. Labs M7-L1 and M7-L3 exercise the error measurements of this lesson.

Related content

Nothing below is required to finish this lesson.

• Before
  M7-T2: System Identification

  Fit the parameters of a robot model from data with least squares, and say when the data are informative enough.

• Lab
  M7-L1: Fit the Dynamics

  Scout's simulator and the real floor disagree: fit friction and drive gain from a log, and check the prediction on new commands.

• Lab
  M7-L3: When the Model Lies

  The plan looks perfect in the model and collides on the floor: measure the model error and shorten the horizon.