M7-T5 — Model Predictive Control with Learned Models

Learning objectives

After this lesson, you should be able to:

• Describe the receding-horizon loop and why replanning corrects model error that an open-loop plan cannot.
• Choose a horizon and a replanning interval using measured success, steps and compute.
• Warm-start the planner from the previous plan, and inflate a collision radius into a safety margin.
• Relate the planning time to the control step, and say what delay and a badly wrong model do to MPC.

The intuition

Steering a car with the eyes closed for three seconds is a bad idea, however good your map. Steering with the eyes open and glancing at the road every tenth of a second is fine even if the map is a bit wrong. MPC is the second way. The model is used to decide what to do next, and the world, not the model, decides where you are after you do it.

The figure shows the one design choice that matters most: the horizon. A very short one (2 steps, 0.5 s) cannot see the pillar coming. Scout carries momentum, so by the time the cost notices it is too late to turn, and 7 of the 20 near-pillar starts crash. From 6 to 16 steps it works every time. At 24 it starts to lose runs again, and slows down: the planner searches a bigger space with the same budget. The longer horizons also take more steps. There is a plateau, not a peak, and the plateau is where you should live.

The mathematics

The receding-horizon loop

1. Observe the state x_t (position and velocity).
2. Plan a sequence A = (a_t, …, a_t+H-1) that minimises the model-predicted cost from x_t (lesson M7-T4).
3. Execute only the first k actions, k = 1 by default.
4. Advance to the new state and repeat from step 1; the rest of the plan is discarded (or used to warm-start the next plan).

The horizon recedes: at each step it starts at the current time. The policy this defines is implicit. There is no network to train, and every control step solves an optimisation. Because the state is re-observed every k steps, an error made in step 3 of the plan is never executed with its full consequences: the plan has been replaced before the error has compounded.

H

Planning horizon in steps (0.25 s each).

k

Replanning interval: actions executed per plan (1 means plan every step).

samples, rounds

Search budget of one plan; model steps per plan = samples x H x rounds.

T_plan, dt

Time to compute a plan and the control step (250 ms in this module).

Why replanning corrects model error

Write the model one-step error as e per step. An open-loop plan of length n accumulates an error that grows with n (lesson M7-T3: roughly proportional to n or faster here). A closed-loop controller sees the true state every k steps, so the error it must live with is only the error over k steps, and it is corrected by feedback: if Scout is slower than planned, the next plan starts from where Scout is. What replanning cannot fix is a model that is systematically wrong in the direction that matters. If the model says "I stop within the margin" and the robot does not, a re-observed state arrives too late.

The cost of replanning

The cost of a plan is T_plan, the time to run samples x H x rounds model steps. The loop is only valid if T_plan is smaller than dt, with margin for sensing and actuation. If the plan takes longer, there are two choices: reduce the search (fewer samples, rounds or horizon), or execute k > 1 actions per plan so that one plan covers k steps. Both lose something, and the sections below measure how much.

Warm starting

The plan made at time t is nearly the right plan at time t + 1 with one step missing. Take the old plan, drop its first k actions, pad the end with zeros (or repeat the last action), and use that as the initial mean of the next CEM search. The initial standard deviation stays wide enough to explore. The plan then starts from a good guess and needs fewer samples.

Constraints and safety margins

The collision penalty uses the pillar’s circle (radius 0.95). Using a larger radius in the planner than in the world, say 1.15, is an inflated constraint: the planner keeps its distance, so an error of up to 0.2 m in where Scout ends up cannot cause a collision. A margin costs path length. This is a design decision about how large model and sensing errors can get, not a guarantee (lesson M5-T5, M10-T1).

A worked example

All numbers are real outputs of Node. The world is the floor (gain 2.2, drag 3.4) unless stated; MPC uses either the floor model or the nominal one (gain 3.0, drag 3.0). The planner is a local copy of RLCore.planCEM (checked to give the same plans) with 48 samples, 6 elites, 3 rounds, horizon 12, initial std 0.7. A run ends when Scout reaches the goal circle (radius 0.5), enters the pillar’s collision circle, or takes 80 steps. Each setting uses 20 starts from RLCore.starts(11, 20, 'all') unless stated, with noise-free dynamics unless stated. I give the number of starts because small samples hide differences.

1. Replanning corrects a wrong model. Replanning at every step with the floor model: 20 of 20 reach the goal (54.6 steps on average); with the nominal model, which is wrong about the world: 20 of 20 (55.1 steps). I repeated this with three different random streams (offsets 0, 1000, 2000): 20 of 20 in all six runs, 54.2 to 55.5 steps. Two other start sets (starts(21, …), starts(22, …)): 20 of 20 for both models, 49.6 to 51.8 steps. Adding process noise of 0.05 to the floor dynamics: floor model 20 of 20, nominal model 19 of 20 with one crash. The wrong model costs almost nothing in this task; the contrast is with the open loop of lesson M7-T4, whose end position was 0.62 m off on average.

2. Horizon sweep (nominal model; the figure uses the 'near' starts, y between 2 and 4, where the pillar is in the way). On 'near': H = 2: 13 of 20 reach, 7 crash, 42.8 steps; H = 4: 19 of 20 (1 crash), 43.6; H = 6: 20, 45.0; H = 8: 20, 48.1; H = 12: 20, 50.6; H = 16: 20, 56.5; H = 24: 19, 63.4 (no crash; the missed run timed out). On 'all': 17, 20, 20, 20, 20, 20, 18 reached out of 20 for the same horizons, with 43.5, 45.4, 48.0, 50.2, 55.1, 59.5 and 64.1 steps. The short-horizon failure is a crash, not the "stuck behind the pillar" behaviour often described. With 0.5 s of lookahead and momentum, Scout cannot see the pillar early enough to turn; I did not see the planner get trapped behind it in these runs. Longer horizons took more steps, which was a surprise. The cause is not model error here (the same nominal model works at H = 12). I think it is the search: with 48 samples x 3 rounds the planner has 24 x 2 = 48 numbers to find at H = 24. Giving it 192 samples and the same 3 rounds restored 20 of 20 (54.1 steps), and 192 samples with 5 rounds gave 20 of 20 in 49.6 steps. I have not tested whether this is the only reason.

3. Replanning interval. Nominal model, H = 12, executing k actions per plan, all starts: k = 1: 20 of 20, 55.1 steps, 55.1 plans per run; k = 2: 20 of 20, 54.9 steps, 27.8 plans; k = 4: 20, 57.6, 14.8; k = 6: 19 (1 crash), 59.9, 10.5; k = 12: 10 of 20 reach it, 2 crash, 71.8 steps, 6.5 plans. Halving the planning effort with k = 2 cost nothing measurable; k = 12 (executing the whole plan) is the open loop of lesson M7-T4 and lost half the runs. With the floor model instead, k = 12 reached 8 of 20, so a perfect model does not rescue executing 12 steps blind here either. My explanation, which I did not test: a 12-step plan only optimises up to its own end, so its later actions are chosen without knowing what comes next, and they are executed without a revision.

4. Warm starting. Nominal model, k = 1. Cold start (initial mean 0): 48 samples x 3 rounds: 20 of 20, 55.1 steps; 24 x 2: 17 of 20, 65.6 steps; 16 x 1: 3 of 20. Warm start (shifted previous plan as the initial mean, zeros appended, initial std 0.7 again): 48 x 3: 20 of 20 in 44.4 steps; 24 x 2: 20 of 20, 46.4; 16 x 1: 20 of 20, 51.7. The 16 x 1 warm planner uses 192 model steps per plan against 1728 for the cold 48 x 3 (9 times fewer) and reached the goal every time. With k = 2 the pattern was the same (warm 16 x 1: 20 of 20, 60.1 steps; cold: 1 of 20). Steps are counted only for runs that reached the goal, so the cold 16 x 1 figure is not comparable (3 runs).

5. Safety margin. Nominal model, noise 0.05, all starts, planner collision radius 0.95 (the true one), 1.15 and 1.35. Reached: 19, 20, 20 of 20. Crashes: 1, 0, 0. The smallest distance to the pillar centre in any run: 0.94 m (the crash), 1.18, 1.38. At noise 0.15, 'near' starts: 20 of 20 reach it for each radius with no crash, even without a margin, but the closest approach was 0.96 m for 0.95, 1.11 m for 1.15 and 1.30 m for 1.35 (means 1.02, 1.21, 1.40), and mean steps 52.1, 54.8, 55.4. A margin of 0.2 m cost about 2.7 steps and bought about 0.2 m of clearance. One crash in 20 runs is thin evidence that the margin prevents crashes; the clearance numbers are the firmer result.

6. Model error against MPC robustness. World: the floor. Model parameters (gain, drag), steady speed = gain/drag, reached of 20: (2.2, 3.4) 0.65 m/s: 20 of 20; (3.0, 3.0) 1.00: 20; (4.0, 2.0) 2.00: 20; (5.0, 1.5) 3.33: 20 (58.1 steps); (8.0, 1.0) 8.00: 20 (65.5 steps). A model that overestimates how fast Scout moves is tolerated. A model that underestimates it fails: (1.5, 5.0) steady speed 0.30: 10 of 20 reach it and 10 crash; (1.0, 8.0) speed 0.13: 2 of 20 and 18 crashes. My reading, not tested: a model that thinks Scout is sluggish believes it can come close to the pillar and still turn away in time, while the real robot has more momentum.

7. Time per plan against the control step. The 48 x 12 x 3 plan took 0.6 ms in Node on this sandbox (1728 model steps), far below the 250 ms step. The 192 x 24 x 5 plan (23 040 steps) took 7.7 ms. The module notes quote about 40 ms for the first; I did not reproduce that, so I do not know which hardware it refers to. A planner running on a robot computer 50 times slower than this sandbox would take 30 ms for the first plan: still inside the 250 ms budget, with 220 ms spare. The unsafe regime is a plan that exceeds the step. Delay. I emulated latency crudely: the plan is computed from the state observed L steps ago, and the first action is applied L steps later (no action before that), without predicting through the actions in flight. Nominal model, all starts: L = 0: 20 of 20; L = 1 (250 ms): 17 of 20 and 3 crashes; L = 2 (500 ms): 10 of 20, 10 crashes; L = 4 (1 s): 6 of 20, 14 crashes. A fix that is standard in practice is to roll the model forward through the commands in flight before planning; I did not test it.

What this simulation simplifies. A point mass, one obstacle, perfect state observation, noise-free or Gaussian process noise, and a model of the right form with wrong parameters. A real learned model is wrong in the form too, and the perception is noisy. The sweeps use 20 starts; differences of one or two runs are not distinguishable from chance.

Engineering implications and limitations

Choosing the horizon

The horizon has to cover the time the robot needs to react, here the stopping or turning time under momentum (the speed time constant is 1/drag, about 0.3 s or just over one step, so the velocity takes roughly four steps to settle; the pillar’s 0.95 m circle is about 1.5 s away at 0.65 m/s). Below it, the planner is myopic. Above it, you pay compute and give the model more room to be wrong. The plateau in the figure runs from 6 to 16 steps, so 8 to 12 is a sensible choice. Read the horizon off the step response of the robot, not off the task length.

Choosing the replan rate

• Replan every step if the planner can finish in the control step. It is the simplest and most robust (example 3: 20 of 20).
• Replan every 2 to 4 steps if the planner is slow. Success dropped from 20 of 20 at k = 4 to 19 at k = 6 and 10 at k = 12; use k well below the horizon.
• Use a faster fallback (a simple controller, or the unexecuted tail of the last plan) between replans, so that a late plan never leaves the robot without a command.

Real-time structure

1. Fix the control step (250 ms here). Measure the worst-case plan time on the target computer, not the average.
2. Set the sampling budget so that the worst case fits in a fraction of the step, leaving time for sensing and actuation.
3. Use the warm start to buy quality for free (example 4: 9 times fewer model steps with equal success).
4. Predict through the delay: plan from the state the robot will be in when the command takes effect.
5. Monitor the planner’s own time and cost: a cost that stops falling across rounds, or a plan that overruns, is a symptom (lesson M10-T5).

Model error, margins and monitoring

MPC is robust to a model that is wrong in a benign direction and not to one that is wrong in a harmful direction (example 6). Test it by perturbing the model parameters in both directions, in simulation, before the robot. Use a margin that matches the measured error, as the model-error study of lesson M7-T3 gives it, and report the clearance as well as the success count (example 5). Check the model against the incoming data while running: when its one-step error grows, the model is being asked about states it never saw, and the safe action is to slow down or stop.

What MPC does not give you

No guarantee of reaching the goal in general (lesson M10-T1 on the sim-to-real gap), no guarantee of safety unless the constraint is enforced by something that does not depend on the model, and no free lunch in compute: the cost of each decision is a full optimisation. In return it needs no policy training, and when the world changes, changing the cost or the model changes the behaviour immediately.

Common misconceptions

Engineering takeaways

1. MPC plans over a horizon, executes the first action (or k), observes, and replans; the model only needs to be right for a few steps.
2. With the nominal or the floor model, replanning every step reached the goal from 20 of 20 starts, in every set of starts and random stream I tried.
3. Horizons 6 to 16 worked; 2 steps crashed (7 of 20 near-pillar starts) and 24 lost runs at a fixed search budget (raising samples to 192 fixed it).
4. A warm start let a plan with 9 times fewer model steps match the cold planner; replanning every 2 steps cost nothing, every 12 lost half the runs.
5. MPC tolerated a model that was too fast but failed on one that was too slow (2 of 20 at steady speed 0.13 m/s), and a delay of 2 steps (500 ms) dropped success to 10 of 20.

References and further reading

Camacho and Bordons is a standard MPC textbook; Nagabandi et al. and Chua et al. combine MPC with learned neural-network models.

• E. F. Camacho and C. Bordons, Model Predictive Control, 2nd edition, Springer, 2004.
• A. Nagabandi, G. Kahn, R. S. Fearing and S. Levine, “Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning”, IEEE International Conference on Robotics and Automation (ICRA), 2018.
• K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models”, Advances in Neural Information Processing Systems (NeurIPS), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next, lesson M7-T6 uses the model a different way: not to plan at decision time, but to generate extra practice for a learner, as in Dyna. It shows the sample-efficiency gain, and the way a biased model can make learning worse.

Related content

Nothing below is required to finish this lesson.

• Before
  M7-T4: Planning with a Model: Random Shooting and CEM

  Plan an action sequence by sampling and refining candidates under a model.

• Lab
  M7-L2: Plan with Random Shooting

  Reach the pharmacy using only a learned model: sample action sequences, score them and refine with CEM.