M7-T1 — Why Learn a Model?

Learning objectives

After this lesson, you should be able to:

• Define a model (a transition function, perhaps with a reward model), and separate learning a model from planning with it.
• Compare model-free and model-based learning by sample efficiency, using counted environment steps rather than slogans.
• Explain model bias: why an optimiser that trusts a slightly wrong model picks plans the model overrates, and how replanning limits the damage.
• Choose between a physical model with few parameters and a black-box model, given the data and the structure you know.

The intuition

Think of a driver who has never been on a particular icy road. One way to learn is to drive it a thousand times and remember which steering works where; that is model-free learning, and it is slow because every lesson costs a trip. Another way is to push the car gently a few times, learn how much it slides, and then imagine routes in your head, choosing the one that keeps clear of the ditch. That is model-based learning. The imagining is free; the trips are not.

The price is that the imagined road is only as good as what you learned about the slide. If you believe the car brakes harder than it does, the route you pick will be the one that brakes latest, because that is the route your belief rates highest. The optimiser is drawn to exactly the places where your model is most flattering.

The figure counts the first kind of cost for Scout. PPO from Module 6 used between about 29,000 and 55,000 simulator steps per run, and three of ten runs never reached a training success of 0.8. The model, in contrast, was fitted from 200 transitions, which is 50 seconds of driving. The comparison is not like for like (the section on honesty below says why) but the gap is two orders of magnitude.

The mathematics

What a model is

A model of an environment is anything that predicts what happens next. In reinforcement learning (Sutton and Barto 2018, chapter 8) it is a pair of functions: a transition model that maps a state and action to the next state (or to a distribution over next states), and a reward model that scores the step. For Scout with momentum, the state is s = [x, y, v_x, v_y], the action is a command a in [−1, 1]², and the simulator of this module is

s = [x, y, v_x, v_y]

state: position (m) and velocity (m/s) of Scout

a

action: a velocity command on each axis, between −1 and 1

gain, drag

physical parameters: acceleration per unit command (m/s²) and velocity decay (1/s)

w

process noise, a standard normal draw per axis, scaled by the noise level (0.03 to 0.08 here)

f_θ

a learned model with parameters θ; ŝ′ is its prediction of the next state

reward / cost model

for the reach task the cost is a known function of the state (distance to the goal, a penalty inside the pillar circle), so only f_θ has to be learned

Two jobs must be kept apart. Learning the model is supervised learning: the data are transitions (s, a, s′) from the robot, and the fit minimises a prediction error. Planning uses the model, without touching the robot, to choose actions: roll candidate action sequences through f_θ, score them, keep the best (lessons M7-T4 and M7-T5). A model-free method skips both: it maps experience straight to a policy or a value function (modules 3 to 6).

The two families side by side

Sample efficiency and model bias

Sample efficiency is the number of real interactions needed to reach a given performance. A model uses each transition for every future plan, which is why model-based methods are famous for needing far fewer; the PILCO method (Deisenroth and Rasmussen 2011) was designed around exactly this route to data-efficient policy search. The flip side is model bias: the planner searches for the action sequence that is best according to the model, and the best of many candidates is often best because of an error, not because of skill. A model that is unbiased on average can still be systematically optimistic about the plan chosen from it. The remedy is not a perfect model, which does not exist, but keeping the error small where the plan goes, and replanning often (lessons M7-T3 and M7-T5).

Physical or black-box

If you know the structure, put it in. A physical model such as the two-parameter form above has few parameters, so 200 transitions pin it down, and it extrapolates because the structure is right outside the data. A black-box model (a small network) makes no assumption, so it can capture friction, slip and effects you did not think of, but it needs more data and has nothing to say where it has seen none. Lesson M7-T2 fits the physical model; lesson M7-T3 trains a network on the same transitions and compares.

A worked example

Every number below is a real output of Node using RLCore (the shared world), unless marked otherwise. Different seeds differ; where I claim something I report several.

1. How many steps did model-free learning use? I re-ran the PPO trainer of Module 6 (trainPolicyGradient, default settings: 60 iterations of 12 episodes) for seeds 1 to 10, counting the environment steps it took. Whole runs used 45,732, 29,692, 54,531, 32,886, 46,750, 54,301, 50,400, 52,938, 46,227 and 35,028 steps. The held-out success of the final policies was 0.92, 0.80, 0.00, 1.00, 1.00, 0.04, 1.00, 0.00, 1.00, 1.00, the same ten numbers as in lesson M6-T7. Seven seeds reached a training success of 0.8 (an iteration of 12 episodes with at least 0.8 reaching the goal) after 38,480, 17,906, 5,254, 30,048, 42,386, 31,657 and 12,375 steps (median 30,048, mean 25,444); seeds 3, 6 and 8 never did. At 0.25 s a step, 30,048 steps is 7,512 s of robot time, over two hours.

2. How many did the model use? I collected 200 transitions from the real-floor dynamics (RLCore.DYN.floor: gain 2.2, drag 3.4) with process noise 0.05: random commands, with Scout reset to a random start every 20 steps (10 resets). That is 200 × 0.25 s = 50 s. The fit is RLCore.ridgeFit on rows [v, a] with target v′ − v (derived in lesson M7-T2). Over five different data sets (seeds 1 to 5), the fitted (drag, gain) were (3.414, 2.200), (3.352, 2.228), (3.385, 2.193), (3.363, 2.211) and (3.361, 2.212), against the truth (3.4, 2.2).

3. Does it work? I planned with each fitted model by model predictive control: at every step, planCEM with horizon 12, 48 samples, 6 elites and 3 CEM iterations, execute the first action, replan. On the noisy floor (noise 0.05), 10 starts per data set, Scout reached the goal in 50 of 50 episodes with no crash, taking 51.3, 53.4, 52.1, 53.4 and 48.2 steps on average for the five models (51.7 overall). No policy was trained at all.

4. The model-bias effect, measured. I planned one open-loop 12-step action sequence with planCEM from each of 20 starts (momentum zero), then compared the cost the model predicted with the cost the sequence earned when executed on the noiseless floor. The cost is planCost (sum of distances to the goal, penalties, a small action cost). The nominal simulator (gain 3.0, drag 3.0), which is wrong about the floor, predicted a mean cost of 74.3 for its chosen plans, and they really cost 78.6: a gap of 4.4, with the final position of the 12-step plan wrong by 0.60 m on average. The fitted model and the exact floor model both predicted 78.6 and earned 78.6 (gap 0.0 to the printed digit). To check that it is the optimising that picks the flattered plans, I also scored 20 random action sequences under the nominal model: predicted 87.7, real 87.6, a gap of −0.2. Errors in random sequences cancel; errors in a chosen sequence do not, because the chosen one is the one the error favoured.

5. Closing the loop repairs it. The same nominal model, replanned at every step, still reached the goal from 20 of 20 starts in the floor world (the test in tests/learning_core.test.mjs). Replanning corrects a wrong model, but only because the error here is mild and Scout can see its true state after each step. A larger error, or a long plan without feedback, is the situation of lab M7-L3 (when the model lies).

What this simulation simplifies, and what the comparison does not show. The PPO runs are in the Module 1 to 6 reach task (positions move by the command, no momentum), the model-based runs are in the momentum world. These are two related tasks, not the same one, and PPO was not given the momentum world; a PPO agent would probably need more steps there, but I did not run it, so I cannot give a number. "Steps to 0.8" is measured on the exploring policy during training. The model-based side also spends compute (1,728 model steps for each decision) that the policy network does not, and it was handed a cost function. And the model class was exactly right: the world is the form that was fitted. With a wrong form, the story changes (lesson M7-T3).

Engineering implications and limitations

When each family wins

• Model-based wins when real data are expensive (a physical robot, a hospital floor), the dynamics are smooth and low-dimensional, and a cost function can be written down. Adaptation is cheap: new goal, new cost, same model.
• Model-free wins when the dynamics are hard to model (contact, deformable objects, perception in the loop), when simulation is plentiful, and when the cost of a decision at run time must be tiny.
• Mixtures are common. Learn a model, use it to plan or to generate extra training data for a policy (lesson M7-T6), or use a policy to propose the plans.

Treat this as guidance, not a theorem. I have shown one task where model-based was far more sample-efficient; that is not evidence for every task.

Before you trust a learned model

1. Write the structure you know. Mass, drag, motor gain, delay: if the physics is known to be of a form, fit the form (lesson M7-T2).
2. Check the data. Did the commands excite every direction the planner will use? A model fitted only to gentle commands says nothing about full ones (lesson M7-T2 and M7-T3).
3. Validate on the horizon you plan over, not only one step ahead: errors compound (lesson M7-T3), and a plan is a multi-step object.
4. Replan. Use the model for short plans that are executed for a few steps and corrected by the next observation (lesson M7-T5).
5. Keep a record of where the data came from. The model is valid on that distribution; a planner will leave it (cross-reference lesson M3-T3, distribution shift).

What to log

Number of real transitions, the commands used, noise level and resets; one-step and multi-step prediction error on held-out data; predicted against real cost of executed plans (the gap in example 4 is a cheap, direct monitor of model bias). The labs of this module, M7-L1 (fit the dynamics) and M7-L3 (when the model lies), turn each of these into an exercise.

Common misconceptions

Engineering takeaways

1. A model is a transition function (and perhaps a reward function). Learning it is supervised learning on (s, a, s′); planning uses it to choose actions.
2. Sample efficiency is a count: 200 transitions (50 s) fitted a Scout model that planned to the goal 50 of 50 times, while the PPO runs used 29,000 to 55,000 steps, on a related task and a model of the right form.
3. Model bias: the optimiser is drawn to the plans the model overrates (predicted cost 74.3, real 78.6 with the nominal model, no gap for random sequences). Keep the model accurate where the plan goes, and replan.
4. Use a physical model when you know the structure, a black-box model when you do not, and validate either on the horizon you plan over, on data like the plan will visit.
5. The advantage is paid for in compute and in model error, so report both next to the sample counts.

References and further reading

Sutton and Barto’s chapter 8 is the standard textbook treatment of models, planning and learning; PILCO is the early demonstration of data-efficient learning with a learned model; PETS shows probabilistic neural-network models matching model-free methods on benchmarks.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018, chapter 8 (planning and learning with tabular methods).
• M. P. Deisenroth and C. E. Rasmussen, “PILCO: A Model-Based and Data-Efficient Approach to Policy Search”, Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.
• K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models”, Advances in Neural Information Processing Systems (NeurIPS), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lesson M7-T2 fits the two physical parameters by least squares, derives the normal equations by hand, and shows how data, noise and commands decide how well they are known. Lesson M7-T3 then asks how wrong a fitted model is after twelve steps.

Related content

Nothing below is required to finish this lesson.