M8-T4 — Conservative and Implicit Methods

Learning objectives

After this lesson, you should be able to:

• Describe the four ways of staying close to the data (policy constraint, value penalty, implicit evaluation, weighted regression) and what each avoids.
• Write the target of the count-based penalty mode and the expectile of a small sample by hand, and say what the CQL regulariser and the expectile do.
• Report, with 20-seed numbers, how an offline policy depends on behaviour epsilon, dataset size and penalty weight, including when conservatism costs nothing and when it costs a lot.
• Explain why the penalty weight cannot be validated without deployment, and what a staged rollout does about it.

The intuition

Take a ward log recorded by a nurse who almost always uses the same corridor. The log contains thousands of steps along that corridor and a handful of steps anywhere else. An optimistic learner asks, for the state next to the stairwell edge, "what is the value of stepping toward it?", finds no data, and answers with whatever its table or network happens to output. If that is high, the policy goes there. A conservative learner treats "no data" as "no evidence that this is good" and gives such actions a low value. Both learners see the same log. They differ in what they assume about the part of the world the log does not describe.

The figure shows the effect on the tabular ward, with the data generated by the optimal policy mixed with random actions at three levels (behaviour epsilon 0.2, 0.5, 0.9, 60 logged episodes). With narrow coverage the optimistic backup gives a policy worth −2.00 at the start, much worse than the nurse’s own policy. Restricting the backup to actions the log contains gives 0.84, and a penalty on rarely seen pairs gives 1.85. With broad coverage the three are close (1.71, 2.29, 2.36), and the cost of being cautious is not visible. None of them reaches the optimum, 3.31.

The rest of the lesson is a closer look at how this is done, how much it costs, and how little we can verify.

The mathematics

The shared idea

The offline setting of lesson M8-T1 gives a fixed dataset D of transitions (s, a, r, s'), produced by a behaviour policy β. The error of lesson M8-T3 arises in the Bellman target r + γ max_a' Q(s', a'): the max ranges over all actions, including those for which Q(s', a') is not supported by any datum. There are four ways to keep the max honest.

1. Constrain the policy to the data (BCQ, TD3+BC)

BCQ (Fujimoto, Meger and Precup, 2019) learns a generative model of the actions in the dataset at each state, proposes a few candidate actions from it, lets a small learned perturbation adjust them within a limit, and then takes the candidate with the highest Q. Both the target and the final policy are then restricted to actions near those in the data. My 'seen' mode below is a tabular cousin of the idea: the max in the target ranges only over the actions that appear in the data at s'. It is a simplification and not BCQ.

TD3+BC (Fujimoto and Gu, 2021) is deliberately simple. It takes the TD3 actor loss, which maximises Q(s, π(s)), and adds a behaviour-cloning term that penalises the squared distance from π(s) to the logged action a. The Q term is divided by its average magnitude so that the balance between the two terms does not depend on the scale of the rewards; the authors report one weight, which I believe is 2.5, working across their benchmarks, but the exact constant is a detail you should check in the paper. The message of that paper is that a small change to an off-policy algorithm gets competitive results on the standard benchmarks, which makes it a good first baseline.

2. Penalise the value of unsupported actions (CQL)

Conservative Q-learning (Kumar et al., 2020) keeps the Bellman error and adds a regulariser that lowers Q on actions chosen by the learner’s own policy or by a broad distribution, and raises it on the actions in the data. In its simplest form, for a weight α:

The log-sum-exp is a soft maximum over actions; subtracting the average Q of the logged actions leaves a net push on actions that look better than their logged counterparts. The authors prove that, for a suitable α, the learned value of a policy lower-bounds its true value (with high probability, under conditions on the data and the function class); a lower bound is what you want if you are about to deploy. I state the result at that level and do not reproduce its conditions. The weight α is the main hyperparameter, and the next sections come back to it.

3. Never evaluate unsupported actions (IQL)

Implicit Q-learning (Kostrikov, Nair and Levine, 2022) avoids the max over actions entirely. It trains a state-value network V(s) by expectile regression on the Q values of actions in the dataset, and uses V(s') in the Q target, so Q is only ever evaluated at (s, a) pairs from D. The expectile loss is asymmetric squared error:

For τ = 0.5 this is ordinary squared error and V(s) is the mean of Q over the logged actions at s. As τ grows toward 1, positive errors (Q above V) are weighted more heavily than negative ones and V(s) moves toward the upper end of the Q values in the data, an approximation to the best logged action at s, while never querying an action that is not logged. The policy is then extracted afterwards, typically by advantage-weighted regression (below). I describe what the expectile does; I do not claim that V converges exactly to the maximum, and the paper’s own argument is about the limit τ → 1 under assumptions I do not reproduce.

4. Weight the logged actions by how good they were (AWR)

Advantage-weighted regression (Peng et al., 2019) fits the policy by supervised learning on the logged actions, each weighted by an exponentiated advantage:

A positive advantage raises the weight of an action, a negative one lowers it, and the temperature β sets how selective this is. The policy can only put weight on actions that are in the data, so it stays close to the behaviour by construction. It is a behaviour-cloning loss with a vote on which examples to imitate (lesson M3-T2 for cloning).

D

The fixed dataset of transitions (s, a, r, s') from the behaviour policy.

n(s,a)

How many times the pair (s,a) appears in D (the tabular stand-in for how well supported it is).

penalty

Weight of the count-based pessimism; mode penalty subtracts penalty / sqrt(n(s,a)) from every target.

α (CQL)

Weight of the CQL regulariser; not the same symbol as the step size in the tabular experiments.

τ

Expectile level in (0, 1); 0.5 is the mean, near 1 is close to a maximum.

β (AWR)

Temperature of the advantage weights exp(A / β).

The two tabular modes used in the experiments

All experiments below call the RLCore.offlineQ function of this track, which sweeps the log in a fixed order, 80 sweeps, step size 0.3, discount 0.95. Its modes:

• naive: target r + γ max over all four actions at s'. Entries that the data never update keep their initial value q0. With q0 = 10 these entries stand for an optimistic extrapolation (the tabular stand-in for an overestimating network); with q0 = 0 they are neutral.
• seen: the max ranges only over actions that appear in D at s' (a behaviour-constrained backup).
• penalty: naive backup (the max still ranges over all four actions, unseen entries included), used here with q0 = 0, but each target is reduced by penalty / sqrt(n(s,a)). A pair seen once is lowered by the full weight, a pair seen 25 times by a fifth of it. This is not the CQL regulariser: it acts on the target, uses counts and has no soft maximum. It captures the same direction of bias: the less supported a pair is, the less its value is trusted.

A worked example

Every number is a real output of Node, from RLCore.gridworld(RLCore.layouts.ward, {slip: 0.2, hazard: 'terminal', stepReward: -0.1, goalReward: 10, hazardReward: -10}), discount 0.95 (optimal value at the start 3.31 by value iteration). The behaviour policy is the optimal policy with probability 1 − ε and a uniformly random action with probability ε, so ε controls coverage. I log N episodes with RLCore.collectDataset, learn offline with RLCore.offlineQ and compute the exact value at the start of the greedy policy with RLCore.evaluatePolicyDP. Every cell is the mean over 20 seeds (SLAMCore.rng(1..20), one dataset per seed). The 95% intervals are Student t over the 20 seeds.

By hand: the penalty target and the expectile

Penalty target. A logged transition has r = −0.1 and the best value at s' is 2.0. With γ = 0.95 the unpenalised target is −0.1 + 0.95 × 2.0 = 1.80. With penalty weight 1 and n(s,a) = 4 the target is 1.80 − 1/√4 = 1.30. For n(s,a) = 1 it is 0.80, and for n(s,a) = 25 it is 1.60. A pair seen once is held 0.8 below its unpenalised target, a well-supported pair is barely touched.

Expectile. Take two Q values at one state, 0 and 10. The expectile m solves τ (10 − m) = (1 − τ) m, so m = 10 τ: for τ = 0.5 it is 5.0 (the mean), for τ = 0.7 it is 7.0, for τ = 0.9 it is 9.0, and for τ = 0.99 it is 9.9 (computed numerically too). For five values (1, 2, 3, 4, 10) the expectiles are 4.00, 5.26, 7.69 and 9.71 at τ = 0.5, 0.7, 0.9, 0.99, against a mean of 4 and a maximum of 10. Raising τ moves V from the mean toward the best logged value without ever evaluating anything that is not in the sample.

AWR weights. With β = 1, an advantage of +1 gives a weight exp(1) = 2.72 and −1 gives 0.37; with β = 3 the same advantages give 1.40 and 0.72, a much flatter vote. An advantage of +5 at β = 1 gives 148.4, which is why the weights are normally clipped.

Experiment 1: coverage (N = 60 episodes)

Behaviour value is the exact value of the logging policy itself, for reference: 2.70 (ε = 0.1), 2.04 (0.2), −0.09 (0.5), −2.03 (0.9). A seed’s dataset has 952 transitions at ε = 0.2 (seed 1) and 4,225 at ε = 0.9.

Three observations. First, the optimistic naive backup is bad at every narrow coverage: −2.00 is far below the behaviour policy’s own 2.04 at ε = 0.2, and much more data of the same narrow kind (1,000 episodes, below) does not fix it. Second, the behaviour-constrained and penalised backups are much better, but the intervals overlap between 'seen' and 'penalty' at ε = 0.5 and 0.9, so I cannot claim an ordering between them there; at ε = 0.2 the gap (0.84 against 1.85) is larger though the intervals still touch. Third, near-expert data (ε = 0.1) shows the limit of the whole approach: the logging policy itself is worth 2.70, and every offline learner is below it (0.43 to 1.33) with 60 episodes. If the data are nearly the policy you want, copying it (lesson M3-T2) is a strong baseline that none of these beats on this log.

The naive learner with q0 = 0 gave exactly the same value as 'seen' in every cell I ran, and I did not investigate why beyond noting that it was identical. A plausible reason (unchecked) is that unseen entries at 0 rarely beat the learned values at the states that matter, since values near the goal are positive. The point stands: the damage in the first column is caused by the optimistic initialisation of unseen entries, which is the tabular stand-in for a network that extrapolates upward, not by the max operator alone.

The initial value matters for the conservative modes too. At ε = 0.2 and N = 60 (20 seeds): 'seen' with q0 = 10 scored −0.16 against 0.84 with q0 = 0, and the penalty mode with q0 = 10 scored −2.00, the same as naive, because the penalty lowers each target but still takes the max over optimistic unseen entries. Only with a neutral q0 = 0 does the penalty help (1.85). A penalty on logged pairs does not protect against values the log never touches. For comparison, an online Q-learner given the same number of environment transitions scored −1.71 in this setting. These values come from the measurements of lesson M8-T3 and the same code; lesson M8-T2 reports that at ε = 0.1 the logs cover only 48% to 50% of the optimal pairs, which is why every offline learner struggles there.

Experiment 2: dataset size

With broad coverage the naive learner needs enough data to fill the table (−0.35 at 20 episodes, 2.51 at 1,000) while the conservative ones are already at 2.64 and 2.68 with 20. With narrow coverage, 50 times more data (1,000 episodes against 20) moved the naive learner from −2.01 to −1.82: more data of the same narrow kind does not close a coverage gap. At 1,000 episodes with broad coverage all three agree to two decimals at 2.51, below the optimum 3.31. I checked one cause: with a smaller step size (0.05) and 400 sweeps the same cell reached 2.71, so part of the remaining gap is the noise of a constant step size in a slippery world and part is the finite sample, and I did not separate them further.

Experiment 3: the cost of conservatism and the penalty weight

If conservatism costs something, it shows where the data are broad and the penalty is strong. For ε = 0.9 and N = 60 the penalty 1 gave 2.36 against 2.29 for the unpenalised ones: no measurable cost at that weight. The penalty weight sweep (mode penalty, N = 60, mean of 20 seeds):

So the cost of too much conservatism is real but only visible in one corner: at ε = 0.2 a weight of 10 fell to 1.02 from 2.25 at weight 5, because a large penalty also depresses pairs that are on the good path but were seen few times. At ε = 0.5 the curve is not monotone (2.34 at weight 1, 1.96 at 5, 2.42 at 10), and with intervals of ±0.6 or more per cell I would not read a trend into it. The more important observation is that the best weight differs by dataset: 5 at ε = 0.2, about 1 at 0.5 and 10 (the largest I tried) at ε = 0.9.

What this simulation simplifies. A table with 23 non-terminal states, a count as the measure of support, 80 full sweeps, one layout, the optimal policy as the base of the behaviour. The deep versions have the same logic and different failure modes (function approximation also generalises between supported and unsupported actions). Do not read the numbers above as a ranking of BCQ, CQL or IQL: I did not run them.

Engineering implications and limitations

Choosing the hyperparameters without an online test

Every method above has a knob that sets how conservative it is (the CQL weight, the TD3+BC balance, the IQL expectile and AWR temperature, my penalty weight), and the sweep above shows that the best value depends on the dataset. In online RL you tune by running the policy. Offline you cannot: the point of offline learning is that you do not run the policy. The options, in order of how much I trust them:

• Off-policy evaluation (OPE) estimates the value of a new policy from logged data, by importance sampling, a fitted value function or a combination. It suffers the same coverage problem as the learner: it is least reliable exactly where the new policy leaves the data. Importance-sampling estimates have a variance that grows with the length of the episode and with the mismatch between policies. Using the learner’s own Q to score itself is circular. I know of no method that gives a dependable estimate of a policy’s real value from logs alone in a new, partly uncovered region; this is an open research problem and you should treat any single OPE number as a rough ranking tool.
• Sanity checks on the log: how far does the learned policy’s action differ from the logged action at each logged state (a small distance is the conservative direction); what fraction of the learned policy’s decisions lie in well-supported (state, action) pairs (lab M8-L2 measures this on the ward)?
• Compare against behaviour cloning trained on the same log (module 3). In experiment 1 the cloned logging policy at ε = 0.1 beat every offline learner. If your offline policy is not clearly better than the clone in the logs, deploy the clone.
• Compare across several weights and seeds, and prefer a setting that is stable over one with a high mean. Report the spread: the intervals in the tables above are wide.
• A small, controlled online test on the real robot or a validated simulator, which is a deployment in the sense of the next section. Every method needs one at the end.

The data side

• Keep the logging policy’s exploration, even when you would like to turn it off: epsilon-style randomness in the log is what makes experiment 1 work at ε = 0.5 and not at 0.1. (Randomness during logging must itself be safe; see lesson M5-T5.)
• Record the behaviour policy’s version, any change in action noise, and the number of transitions per (state, action) region, so that you can compute coverage later (lesson M8-T2).
• Prefer a mixture of datasets (several operators, several policy versions) to a single narrow source.
• Report both the value you estimated and the data it rests on.

Safety

An offline policy is still a policy. Its training never touched the robot, and that is an advantage for the training and none at all for the deployment: its behaviour outside the data is exactly what you could not measure. It needs the same staged rollout as any other policy: a simulator or bench check, a supervised trial with a human ready to stop it, action limits and monitors on the real robot, and a clear rule for going back to the old controller. Lesson M10-T5 gives the checklist; the constraint and cost framing of lesson M5-T5 applies to what the learner is allowed to do in the real world. A conservative method lowers the chance of a surprise outside the data and does not remove it.

Which method when

Common misconceptions

Engineering takeaways

1. Conservative offline methods stay close to the data in four ways: constrain the policy (BCQ, TD3+BC), penalise unsupported Q-values (CQL), avoid evaluating unsupported actions (IQL expectiles), or weight logged actions by advantage (AWR).
2. On the ward with narrow coverage (ε = 0.2, 60 episodes), mean exact value 20 seeds: optimistic naive −2.00, behaviour-constrained 0.84, count-penalised 1.85, against an optimum of 3.31; with broad coverage (ε = 0.9) the three are within 0.7 of each other.
3. More data from a narrow behaviour does not fix the gap (naive: −2.01 at 20 episodes, −1.82 at 1,000), and the cost of conservatism was not measurable with broad coverage except with an excessive penalty.
4. The best penalty weight differs between datasets and cannot be validated by an online test you do not have; OPE is unreliable outside the data, so choosing it is an open problem.
5. An offline policy needs the same staged rollout, monitors and rollback as any other (lesson M10-T5), and compare it with behaviour cloning of the same log before believing it.

References and further reading

Primary papers for CQL, IQL, BCQ, TD3+BC and AWR are listed; the survey by Levine and colleagues is a good overview. I describe each method in words and do not reproduce their algorithms or hyperparameters.

• A. Kumar, A. Zhou, G. Tucker and S. Levine, “Conservative Q-Learning for Offline Reinforcement Learning”, Advances in Neural Information Processing Systems (NeurIPS), 2020.
• I. Kostrikov, A. Nair and S. Levine, “Offline Reinforcement Learning with Implicit Q-Learning”, International Conference on Learning Representations (ICLR), 2022.
• S. Fujimoto, D. Meger and D. Precup, “Off-Policy Deep Reinforcement Learning without Exploration”, International Conference on Machine Learning (ICML), 2019.
• S. Fujimoto and S. S. Gu, “A Minimalist Approach to Offline Reinforcement Learning”, Advances in Neural Information Processing Systems (NeurIPS), 2021.
• X. B. Peng, A. Kumar, G. Zhang and S. Levine, “Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning”, arXiv:1910.00177, 2019.
• S. Levine, A. Kumar, G. Tucker and J. Fu, “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, arXiv:2005.01643, 2020.
• W. K. Newey and J. L. Powell, “Asymmetric Least Squares Estimation and Testing”, Econometrica 55(4), 1987 (the expectile).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Conservative methods decide what to trust in a fixed log. The next lesson (M8-T5) is about how the log is stored and reused: replay buffers, prioritisation, relabelling and the cost in robot-hours behind each stored transition.

Related content

Nothing below is required to finish this lesson.

• Before
  M8-T3: Extrapolation Error in Offline RL

  Explain how a value function overestimates actions that are absent from the data, and the harm it does.

• Lab
  M8-L3: Compare Online and Offline Policies

  Train one policy online and one from a fixed log, and explain why the offline one fails in states the log never showed.