M7-T6 — Dyna and Model-Based Reinforcement Learning

Learning objectives

After this lesson, you should be able to:

• Write the Dyna-Q loop and say what each of its three parts (direct update, model learning, planning updates) does.
• Quantify the sample-efficiency gain from planning steps, and say what it costs in compute.
• Explain how a biased or deterministic-by-construction model degrades learning, with the measured example.
• Describe short model rollouts (MBPO), learned models inside a policy-gradient loop, and world models, and list the conditions in which model-based RL fails.

The intuition

Suppose you walk a corridor once and find a fire door. A model-free learner updates what it believes about that one step and moves on. A model-based learner remembers what happened, and between steps replays it in its head: "from there, going right, I got to the door." The second learner extracts more from the same single walk. That is Dyna.

The figure shows the effect in numbers. On a gridworld with a stairwell edge (the cliff), plain Q-learning needed 13.9 episodes to find a good policy, on average. With 5 imagined updates after every real step it needed 4.2, and with 20 it needed 2.4. The real experience was cut by a factor of 4 for exactly the same learning rule. It was not free: each real step now costs 21 table updates, which is a cheap thing to buy on a computer and an expensive thing to buy on a robot that must physically move.

The same figure hides the risk. In a world where moves sometimes slip, the same imagined updates replay one unlucky outcome over and over, and learning gets worse, not better. The rest of the lesson is about that trade.

The mathematics

Dyna-Q

Tabular Q-learning from lesson M4-T6 updates Q(s, a) from one real transition (s, a, r, s'). Dyna-Q adds a model and uses it between real steps:

1. Act in the real world with an epsilon-greedy policy; observe (s, a, r, s').
2. Direct update: Q(s,a) ← Q(s,a) + α [r + γ max_a' Q(s',a') − Q(s,a)] (unless s' is terminal).
3. Model learning: store Model(s, a) = (r, s'). In the tabular, deterministic version this is a lookup table that keeps the last outcome seen.
4. Planning: repeat n times: pick a previously visited pair (s, a) at random, read (r, s') from the model, and apply the same Q update.

With n = 0 this is Q-learning. Planning here is not a search: it is more value updates, on simulated experience. The relation to the real-time planning of lesson M7-T4 is that both use a model; Dyna spends the model’s predictions on training a value function, MPC spends them on one decision.

n

Planning updates per real step (0, 5, 20 here).

model

A table from (s, a) to the last observed (r, s', done).

α, γ, ε

Step size 0.5, discount 0.95, exploration 0.1 in these experiments.

When the model is wrong or random

A tabular model that remembers the last outcome is exact when the world is deterministic and the pair has been seen. If the world is stochastic, it replays whichever outcome happened last and treats it as certain: the planning updates then converge to the value of a deterministic world that never existed. With a function-approximation model the errors are of a different kind, a model that is wrong in states it has not seen (lesson M7-T3), and a value function can exploit that error by choosing actions the model flatters.

Short rollouts and policy optimisation (MBPO)

Janner et al. (2019) used a learned model to generate synthetic transitions for an off-policy actor-critic. The key design is to start rollouts at states from the real replay buffer and to keep them short (a few steps), so that model error cannot compound the way it does over long rollouts (the growth shown in lesson M7-T3). They also mix real and model data. The theory they give bounds the return when the model error is small; the practical message is that a short horizon is a lever on model bias.

Learned models inside a policy-gradient loop

A model can supply the rollouts for an on-policy method of module 6 as well: instead of running the robot for the 12 episodes of a PPO batch, run most of them in the model. The gradient is then of the model’s world, so the policy improves for the model and the improvement transfers only as far as the model is right. In practice this is stabilised with the same three tools: short rollouts, an ensemble of models to measure disagreement, and a periodic return to real data. I describe this at the level of the idea; this module has no implementation.

World models

Ha and Schmidhuber (2018) trained a model of an environment from pixels, a compressed latent state and a recurrent dynamics model, and then trained a small controller largely inside the model’s imagined rollouts. The idea that a learned latent model can serve as a simulator for a policy is the one that links this module to the vision-language-action (VLA) and world-model track, where the model predicts images or latent states instead of a 4-number state.

A worked example

Every number is a real output of Node. The code is Dyna-Q written against RLCore.gridworld with the standard layouts of lesson M4-T6: the cliff (8 x 4, start bottom-left, goal bottom-right, a stairwell edge between them that costs −100 and returns Scout to the start) and the ward (9 x 7 with walls and a hazard). Step reward −1, goal +10, discount 0.95, step size 0.5, epsilon 0.1, up to 200 steps per episode, 20 seeds (SLAMCore.rng(1..20)). The model stores the last observed outcome per (state, action). Planning picks one of the pairs seen so far uniformly at random.

"A good policy." After each episode I compute the exact expected return of the greedy policy from the start, using RLCore.evaluatePolicyDP, and count the first episode at which it is at least a threshold. Ward: within 0.5 of the optimum (−1.723). Cliff: the optimum is −0.761; the value of the safe path that keeps away from the stairwell is −2.64, and I used −3 as the threshold so that "good" means the greedy policy reaches the goal without falling.

So 5 planning updates per step cut the real steps by a factor of 3.2 (cliff) and 3.0 (ward), and 20 by 4.3 and 4.9. The returns diminish: 50 updates per step on the cliff took 1.9 episodes and 135 real steps, on the ward 2.3 and 149 (by the stricter optimal-within-0.5 criterion, see below). Where the real data are scarce, the first few planning steps buy most of the gain. The cost: 6, 21 or 51 table updates per real step. These are trivial on a table and not trivial on a neural network trained by gradient steps. One cliff seed with n = 20 never reached the threshold within 300 episodes; its greedy policy still had value −4.33.

A check on the criterion. If I require the policy to be optimal on the cliff (within 0.5 of −0.761), only 18 of 20 seeds with n = 0, 18 of 20 with n = 5 and 16 of 20 with n = 20 got there in 300 episodes. The others settled on the safe path at −2.64. With epsilon 0.1 the shortest path along the edge is risky to explore, so a safe policy once found is rarely improved; more planning does not fix a path that is never tried. Planning accelerates what the real data show and does not invent data.

The same method in a stochastic world. Slip 0.2: with probability 0.2 the move turns 90 degrees to one side or the other (as in gridworld({slip: 0.2})). The optimum values are −10.27 (cliff) and −9.32 (ward). The exact value of the greedy policy from the start, averaged over 20 seeds, after 150 episodes:

At 40 episodes the pattern was the same (cliff n = 0: −13.9, n = 5: −32.1, n = 20: −49.6; ward −13.0, −14.0, −15.7). Planning made the policy worse. The model keeps one outcome per (state, action), so a single slip into the stairwell is replayed 20 times as though it were certain, and a single lucky run past it is replayed as certain too. The value function is trained on a world with no randomness in it, and the greedy policy follows that world’s optimum, not the real one. I attribute the gap to this mechanism; I did not test other factors, such as the large step size of 0.5, which also hurts plain Q-learning in noise (its −15.25 is 5 below the optimum), and the averages are over 20 seeds with a few very bad policies.

What this simulation simplifies. Tabular, tiny state spaces, a model that is a lookup table, and a model-free baseline with an untuned step size. The sample-efficiency gain on a real robot depends on how accurate a model can be learned from how little data. I tested only these two layouts.

Engineering implications and limitations

Practical recipe

1. Start from a working model-free learner (lesson M4-T6 or M6-T5); Dyna is an addition, not a replacement.
2. Fit the model on real transitions with a held-out check (lesson M7-T3): report its error as a function of horizon.
3. Generate model data from real start states with short rollouts, a few steps, and mix them with real data in the replay buffer.
4. Increase the ratio of model to real updates slowly, and compare against the model-free baseline at the same real-sample count, on several seeds.
5. Evaluate on the real robot or the real simulator, never on the model.

Keeping bias under control

• Short rollouts: error compounds with rollout length, so cap it. MBPO (Janner et al., 2019) does exactly this.
• Ensembles: train several models and treat their disagreement as uncertainty; reject or down-weight transitions where they disagree (as in Chua et al., 2018, for planning).
• Model the noise: predict a distribution (mean and variance), not a point; the table in the example fails because its model has no way to say "this move slips 20% of the time".
• Keep real data in the mix, so the value function never trains on model data alone.
• Monitor the gap between returns predicted in the model and returns obtained in the world. When they separate, the learner is learning the model, not the task (lab M7-L3).

When model-based RL fails

World models and the VLA track

Ha and Schmidhuber (2018) showed that a latent-state model learned from images can host a controller’s training. Models that predict the future of a robot’s camera or latent state, conditioned on actions or language, are the base of much current work on vision-language-action models; that is the subject of the VLA track and is outside what this lesson can measure. Everything above (bias, compounding error, short rollouts) applies to them with larger numbers.

Common misconceptions

Engineering takeaways

1. Dyna-Q alternates real steps with n imagined updates replayed from a learned model; n = 0 is Q-learning.
2. On the cliff, episodes to a good policy fell from 13.9 (n = 0) to 4.2 (n = 5) and 2.4 (n = 20); on the ward from 19.3 to 7.0 and 3.5 (mean of 20 seeds).
3. A last-outcome model in a stochastic world made learning worse: cliff value −30.92 at n = 5 against −15.25 for Q-learning (optimum −10.27).
4. Short model rollouts from real states, ensembles, probabilistic models and mixing real data are the standard defences against model bias.
5. Model-based RL fails with compounding error, sparse rewards the model has never seen, stochastic dynamics and model exploitation; world models extend the idea to images and link to the VLA track.

References and further reading

Sutton introduced Dyna; Sutton and Barto’s textbook has the Dyna-Q algorithm; Janner et al. and Ha and Schmidhuber are the model-based references cited above.

• R. S. Sutton, “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming”, Proceedings of the 7th International Conference on Machine Learning (ICML), 1990.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.
• M. Janner, J. Fu, M. Zhang and S. Levine, “When to Trust Your Model: Model-Based Policy Optimization”, Advances in Neural Information Processing Systems (NeurIPS), 2019.
• D. Ha and J. Schmidhuber, “World Models”, arXiv:1803.10122, 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes the model-based module. Labs M7-L1 to M7-L3 practise fitting a model, planning with it and diagnosing its lies. Module 8 turns to learning from fixed datasets, where the same question appears from the other side: what the data never show cannot be learned, with a model or without one.

Related content

Nothing below is required to finish this lesson.

• Before
  M7-T5: Model Predictive Control with Learned Models

  Replan at every step with a learned model, and say what the horizon trades off.

• Lab
  M7-L3: When the Model Lies

  The plan looks perfect in the model and collides on the floor: measure the model error and shorten the horizon.