M8-T1 — Learning from Fixed Datasets

Learning objectives

After this lesson, you should be able to:

• Define offline (batch) learning and say where robot logs come from.
• Compare online RL, offline RL and imitation by what each needs and what each assumes.
• State the principle that the log bounds what is learnable, and give one example of a task the log cannot teach.
• Distinguish offline policy evaluation from offline policy improvement, and explain why the first is also hard.

The intuition

Picture a hospital robot fleet that has been running for a year. Every night the logs are uploaded: observations, actions, rewards, the next observations. Nobody wants to send a robot into a corridor full of patients to "explore", yet the log holds millions of transitions. Offline learning asks what policy can be extracted from that record alone.

The figure shows the three settings you have met as flows of data. In online RL the loop closes: the learner changes the policy, the policy produces new experience, and the experience changes the learner. Whatever the learner is unsure about, it can go and check. In imitation the data are demonstrations by a good actor and the task is to copy them; no reward is needed, and the distribution shift of lesson M3-T3 is the main enemy. In offline RL the data have a reward and are not from a good actor, necessarily, but there is no loop: the learner cannot check anything.

The missing arrow has one consequence that this whole module is about. An online learner that wrongly believes that "pushing the door left" is great will push it, observe the result and correct itself. An offline learner that holds the same wrong belief will keep it, because nothing in the log contradicts a belief about something that is not in the log. The log can be large and the belief still unchecked.

The mathematics

The setting

A dataset D is a list of N transitions collected in the world by some behaviour policy β (which may be one policy, many, a human, or a mix, and which may be unknown):

(s, a, r, s′, d)

State, action, reward, next state and a flag that the episode ended; the transition format of lesson M4-T6.

β

The behaviour policy that wrote the log. Its choices decide which (s, a) pairs exist in D.

J(π)

The return of the learned policy once deployed; the number that matters and that the log cannot measure directly.

N

The number of transitions. More of them helps only if they are about the right states and actions (lesson M8-T2).

Where robot logs come from

• Fleet logs: every robot in production is already writing (observation, action, outcome). They are plentiful, they reflect the real world, and they come from whatever controller was deployed, so they show mostly typical behaviour and few failures.
• Teleoperation: a person drives the robot (lessons M3-T1 and M3-T5). The data are good in the states the person visited and silent elsewhere; the person rarely makes the mistakes a learner will make.
• Earlier controllers: a hand-written controller or an earlier learned policy. Its logs have its quirks, including its blind spots.
• Earlier experiments: replay buffers from past training runs (lesson M8-T5). These are broad and badly aimed: they contain the learner’s early mistakes.

Three settings compared

The principle: the log bounds what is learnable

I call it a principle, not a theorem. It says: no algorithm can reliably know the value of a decision the data say nothing about. If the log never shows the robot turning left in the corridor, then whether turning left is good is a guess, however the guess is made, and a learner that wants to be right must either avoid that decision or be told what to assume. Three things follow, in the order the next lessons take them.

• Coverage decides the ceiling. The best policy the learner can be sure about uses only choices that the log shows (lesson M8-T2).
• The standard backup breaks the principle. The Q-learning target takes a maximum over all actions, including those the log never shows (lesson M8-T3).
• Fixes keep the learner inside the data, or make it doubt what lies outside (lesson M8-T4).

Evaluation versus improvement

Offline policy evaluation (OPE) asks: given a fixed policy π and the log D from a different policy β, estimate J(π)? Offline policy improvement asks: find a π with a high J(π). Improvement needs evaluation, since choosing needs comparing, and evaluation is hard for the same reason as everything else: the log shows only what β did. Three families are used:

• Importance sampling re-weights each logged episode by the ratio of the probability that π would have taken the same actions to the probability that β did. It is unbiased when β covers π and the probabilities are known, and its variance grows quickly with the length of the episode.
• Fitted Q evaluation learns Q for π from the log by the Bellman backup r + γ Q(s′, π(s′)), with no maximum, and reads off the value at the start. It is the policy-evaluation counterpart of lesson M4-T5.
• Model-based evaluation fits a model (lesson M7-T3) and runs π in it, with the model error of that lesson.

I use only fitted Q evaluation below. In all three the estimate of a policy that strays from β rests on data that do not exist.

A worked example

Every number comes from running Node against RLCore. Scout’s ward is RLCore.gridworld(RLCore.layouts.ward, {slip: 0.2, hazard: 'terminal', stepReward: -0.1, goalReward: 10, hazardReward: -10}): 25 open cells, 23 of them non-terminal, discount 0.95. Value iteration gives an optimal value of 3.31 at the start (lesson M4-T4). The behaviour policy is the optimal policy with an epsilon-random action: with probability ε = 0.2 a uniformly random action, otherwise the optimal one. collectDataset logs 60 episodes. The learner is offlineQ: tabular Q-learning over the log (step size 0.3, 80 sweeps), which uses the log and nothing else. Each run is scored by the exact expected return from the start of the greedy policy, with evaluatePolicyDP. Mean of 20 seeds (1 to 20); each seed has its own log.

What the log contains. About 937 transitions (902 to 969). In the log, 85% of the episodes reach the goal and 15% fall into the stairwell, and their mean discounted return is 2.22. That looks like a good log. But only 49% of the 23 non-terminal states appear in it at all: Scout follows the corridor and slips off it only now and then. Of the (state, action) pairs, 46% appear.

Three learners on the same 20 logs (the learned policy’s exact value at the start, optimal 3.31):

The same algorithm, the same log, and a spread from −2.00 to 1.85 that comes only from what the backup believes about actions the log lacks (lesson M8-T3 explains the first row). The value −2.00 is not random bad luck: it is exactly −0.1 / (1 − 0.95), the return of a Scout that never reaches the goal. In all 20 seeds the optimistic learner’s greedy policy never reaches the goal. In the one log I traced (seed 1), Scout stands in the corridor cell below the start and pushes against a wall that its inflated value says is promising (lesson M8-T3 traces it). Deployed for 200 simulated episodes per seed, it reached the goal in 0% of them; the "seen" and penalty learners reached it in 83% of episodes, and fell into the stairwell in 9% and 11%. The naive-0 learner and the "seen" learner returned an identical value in all 20 seeds. I did not investigate why the two coincide.

A comparison with online learning. A tabular online Q-learner (ε = 0.1, step size 0.3, initial values 0, 20 seeds) given the same number of environment steps as the log (937) reaches −1.71 (−1.98 to −1.44), and with 1,500 steps 0.14 (−0.99 to 1.27). So, in this experiment, the log from a near-optimal behaviour is worth something: a good offline learner beats an online learner that starts from nothing with the same budget. That statement is about this log, this learner, and this budget, not about offline learning in general (lab M8-L3 lets you repeat the comparison with your own settings); with ten times more steps the online learner is optimal and the log is capped by its coverage (lesson M8-T2).

Offline evaluation of fixed policies. Fitted Q evaluation (200 sweeps, step size 0.3, unseen entries start at 0), value at the start estimated from the log, mean of 20 seeds, with the true value computed exactly:

The narrow log estimates the policy that resembles β well, and is off by 1.74 for a policy that differs from it in one early move (the estimate is 0.00 while the truth is 1.74, with a confidence interval wide enough to include both). The broad log is closer on every row. Policies far from β are exactly the ones whose estimates are the least trustworthy, and exactly the ones a policy-improvement step wants to propose.

What this simulation simplifies. A tabular world with 23 states, one layout, exact scoring by dynamic programming (which a real robot does not have), one family of behaviour policies, one learner and one step size, and no function approximation. The size of the gaps will differ elsewhere; their direction is what the experiment supports. The online comparison used untuned settings.

Engineering implications and limitations

Before you train on a log

1. Write down what produced the log: policy version, time period, robot, operating mode, who drove. If you cannot, you cannot say what it covers (lesson M8-T2 and the data card of lesson M2-T6).
2. Check that actions and rewards are logged correctly: the action that was actually executed (after safety clipping), the reward as it would be computed today, the end-of-episode flag, and timestamps consistent with the control rate.
3. Split by episode, session and robot, not by row, so that a held-out set is really held out (lesson M2-T6).
4. Measure coverage of the states and actions the policy will meet (lab M8-L2) before any training.
5. Pick a conservative learner first (lesson M8-T4) and compare it with the behaviour policy’s own score, which is the baseline to beat.
6. Evaluate before deployment: offline estimates for a shortlist, then a small, supervised, reversible trial on the real robot, with a fallback (lessons M10-T4 and M10-T5). Offline estimates choose candidates; they do not certify them.

Offline first, online afterwards

A common pattern is to pretrain from a fleet log, then fine-tune online with a small, safe amount of interaction. This helps because the learner starts from a sensible policy, and the online phase can check the beliefs that the log could not. It is not automatic: an offline policy that is confidently wrong can crash the first online episodes, so the online phase needs the same safeguards as any trial (constraints, lesson M5-T5).

The offline setting is not always the right one

Common misconceptions

Engineering takeaways

1. Offline learning produces a policy from a fixed log of transitions, with no further interaction; logs come from fleets, teleoperation, earlier controllers and past experiments.
2. Online RL can check its own beliefs, imitation copies a demonstrator, and offline RL has rewards but cannot try anything: the log bounds what is learnable.
3. On the ward, the same offline learner and the same 20 logs gave −2.00 (optimistic about unseen actions), 0.84 (neutral), and 1.85 (penalised), against an optimum of 3.31.
4. Evaluating a policy offline is hard for the same reason improving it is: policies far from the behaviour policy are estimated from data that do not exist (0.00 estimated against 1.74 true).
5. Check what produced the log, split by session, measure coverage, prefer conservative learners, and always finish with a careful real-robot trial.

References and further reading

The Levine et al. tutorial is the standard survey of offline RL and its open problems. Lange, Gabel and Riedmiller is an earlier survey of batch RL; Sutton and Barto give the Bellman backups used here.

• S. Levine, A. Kumar, G. Tucker and J. Fu, “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, arXiv:2005.01643, 2020.
• S. Lange, T. Gabel and M. Riedmiller, “Batch Reinforcement Learning”, in M. Wiering and M. van Otterlo (eds), Reinforcement Learning: State of the Art, Springer, 2012.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

We said the log decides what can be learned, and measured only one log type. The next lesson, M8-T2, measures what a dataset covers and what it never shows, and how the behaviour policy and the dataset size change the result: with the same ward, the same learners and 12 combinations of dataset size and behaviour randomness.

Related content

Nothing below is required to finish this lesson.

• Lab
  M8-L3: Compare Online and Offline Policies

  Train one policy online and one from a fixed log, and explain why the offline one fails in states the log never showed.