M6-T6 — SAC and Off-Policy Learning

Learning objectives

After this lesson, you should be able to:

• Explain why a replay buffer makes learning off-policy, what it gains in sample efficiency, and what stale data and a moving bootstrap target cost.
• Write the maximum-entropy objective, the soft Bellman backup and the SAC critic and actor losses, and say what each term does.
• Show with a small simulation why the minimum of two critics reduces overestimation, and why it does not remove all bias.
• Compute the entropy of a Gaussian policy, apply the tanh correction to a log-probability, and say when off-policy methods beat on-policy ones and when they fail.

The intuition

Think of two ways to learn a route from a diary. The on-policy learner reads today’s pages once, updates its habits, and burns the diary, because tomorrow's habits make today’s pages slightly misleading. The off-policy learner keeps every diary it has ever written and rereads random pages while it trains. It is right to do so only if the question it asks of a page does not depend on who wrote the page. An action-value does not: if I take action a in state s, and afterwards behave like policy π, what do I collect? The first step is fixed by the page, and only the later steps use the current policy, which the learner can evaluate itself. So Q-functions can learn from old pages, and policy gradients (lesson M6-T2) cannot without corrections.

Reuse has two prices. The old pages cover the states an old, worse policy visited, and the critic is trained on its own predictions, a moving target. SAC pays both prices in a specific way. Entropy keeps the policy exploring, so the buffer does not fill with near-identical trajectories. Two critics and slowly updated target copies keep the bootstrap value from running away. The figure shows the loop.

The mathematics

Replay and off-policy updates

A replay buffer stores transitions (s, a, r, s′, done) from every policy the learner has been. A training step samples a random minibatch and takes a gradient step on the critic, then on the actor. Random sampling also breaks the strong correlation between consecutive steps of one trajectory, one reason deep Q-networks became stable enough to use (Mnih et al., 2015). Three consequences matter.

• Sample efficiency. One transition can be used in hundreds of updates. The ratio of gradient steps to environment steps (the update-to-data ratio) is a setting of its own; a ratio of 1 is typical for SAC, and PPO in this module uses each sample 5 times at most in a batch and then discards it.
• Bias from stale data. The buffer follows the old behaviour distribution, not the current policy. For the one-step action-value that is allowed, because only the first action comes from the page. But the critic is fitted where the buffer has data, and the actor is then pushed toward actions where the critic is large, possibly where the buffer has none. There the critic is an extrapolation. This is the seed of the offline failures of Module 8 (lesson M8-T3).
• A moving target. The regression target y contains the critic itself. Bootstrapping, function approximation and off-policy data together are the "deadly triad" of Sutton and Barto (chapter 11): the three can make values diverge. A target network, a copy of the critic whose weights follow the learner slowly (θ̄ ← τθ + (1 − τ)θ̄ with τ about 0.005), keeps y almost still during a step, so the regression has something stable to fit.

The maximum-entropy objective

Ordinary reinforcement learning maximises expected return. SAC maximises return plus a bonus for the policy staying random:

H(π(·|s))

the entropy of the action distribution at s, in nats. Larger means more random. For a Gaussian with standard deviation σ in each of k independent dimensions, H = k · ½ ln(2πe σ²).

α

the temperature, a positive number that trades reward against entropy. α → 0 recovers the ordinary objective.

Q(s, a)

the soft action-value, defined below; V(s), the soft state value.

Why would a robot want that? Haarnoja et al. (2018) give two reasons: the policy keeps exploring and does not collapse early onto one action, and it learns several ways of achieving a goal where several exist, instead of committing to one. The cost is that the optimum is no longer the pure return optimum: a large α makes Scout trade reward for randomness it does not need.

The soft Bellman backup and the twin critics

The minimum of two critics answers overestimation. A critic is a noisy estimate, and a policy that maximises a noisy estimate seeks out its upward errors, so the target is too high, the next critic is fitted to it, and the error accumulates through bootstrapping. Fujimoto et al. (2018, TD3) proposed training two critics and using the smaller; SAC adopts it. The minimum is biased low, but a low bias is safer than a high one that feeds on itself. The example below measures this.

Squashed Gaussian policies and the reparameterisation trick

Scout’s actions live in [−1, 1]. SAC's actor outputs a mean μ(s) and standard deviation σ(s), draws u = μ + σ·ε with ε ~ N(0, 1), and returns a = tanh(u). This reparameterisation writes the random action as a differentiable function of the network outputs and an independent noise sample (Kingma and Welling, 2014), so the actor loss above can be differentiated through the critic straight to μ and σ. The log-probability of a is not that of u: tanh squeezes the line, which changes the density, and a change-of-variables correction applies:

Compare the policy gradient of lesson M6-T2, which uses the score function ∇log π and needs only the sampled reward. The reparameterised gradient uses the critic’s derivative with respect to the action, so it needs a differentiable Q, and often has lower variance. That is why SAC is a method for continuous actions only.

The temperature

Choosing α by hand is awkward, because a good value depends on the reward scale. Haarnoja et al. (2018, "Algorithms and Applications") make α a learned variable: it is increased when the policy’s entropy falls below a target and decreased when it is above, so the entropy is held near the target. The usual target is −k for a k-dimensional action, which is −2 for Scout. Because the SAC entropy is that of the squashed action, this is a rough guide: for the unsquashed Gaussian, an entropy of −1 per dimension corresponds to σ ≈ 0.089.

A worked example

Every number below was computed by running a few lines of Node (the Monte Carlo used the repository's seeded generator, 200,000 trials, seed 42). None of it is output of a SAC agent: the reference trainer has no SAC.

1. A soft value, by hand. At s′ the policy picks three actions with probabilities 0.5, 0.3, 0.2 and the critic gives them Q = 5, 4, 3. Take α = 0.2.

E[Q] = 2.5 + 1.2 + 0.6 = 4.3. The entropy H = 0.5 × 0.6931 + 0.3 × 1.2040 + 0.2 × 1.6094 = 1.0297 nats. V = 4.3 + 0.2 × 1.0297 = 4.5059, which is the sum of the last column. The randomness is worth 0.206 of value. A deterministic policy that always picked action 1 would have V = 5, higher: the soft value does not say randomness beats greed, it says what the randomised policy is worth.

2. One SAC target. Scout steps with r = −0.05. At s′ the policy samples a′ with log π(a′|s′) = −1.5, and the two target critics say 7.2 and 6.8. With γ = 0.99, α = 0.2:

Using only critic 1 gives y = 7.375; using no entropy term gives 6.682. The minimum lowers the target by 0.396, the entropy bonus raises it by 0.297. A rarely-chosen action (low probability) earns a larger bonus, so the target rewards visiting what the policy usually avoids.

3. The minimum of two noisy critics. Suppose every action has true value 0 and each critic estimates it with independent noise of standard deviation 1. For one action, the mean of a single critic is 0.0 (−0.0006 measured) and the mean of the minimum of two is −0.5642 (theory −1/√π; −0.5608 at 200,000 draws on a second stream). With four actions a policy that takes the best-looking one sees:

The one-critic maximum overestimates by about one noise standard deviation, matching the 1.03 quoted in lesson M4-T6. The minimum cuts that to 0.27: it helps a lot but does not make the estimate unbiased, and for a single action it is biased the other way (−0.56). A bias of just 0.3 per backup matters because bootstrapping adds it up: with γ = 0.99 a constant 0.3 added at every backup becomes 0.3 / (1 − 0.99) = 30 in a value that should be of order 10. This is a worst case, not a measurement of an agent.

4. Entropy of a Gaussian policy. In two independent action dimensions, H = 2 × ½ ln(2πe σ²):

The difference is 2 ln(0.5 / 0.1) = 2 ln 5 = 3.2189 nats: entropy depends only on the log of σ, so shrinking σ five times costs about 1.6 nats per dimension, whatever σ you start from. The first line matches the entropy of 1.452 that RLCore.trainPolicyGradient logs for its default initStd 0.5 (lesson M6-T7). Negative entropy is normal for continuous distributions: it measures density, not the number of outcomes.

5. The tanh correction. Let μ = (0.4, −0.2), σ = (0.5, 0.5), and the noise draw give u = (0.8, −0.6), so ε = (0.8, −0.8). Then a = tanh(u) = (0.6640, −0.5370). log N(u) = −1.0916 (the two Gaussian terms), and Σ log(1 − tanh²u) = −0.9218. So log π(a|s) = −1.0916 − (−0.9218) = −0.1698. Without the correction the log-probability would be wrong by 0.92 nats, and so would the entropy bonus.

What this does not show. No learning curve: I did not train SAC, so there is no claim here about its sample efficiency on Scout's task. The claims about SAC's behaviour are those of its authors on simulated locomotion benchmarks, which differ from this task.

Engineering implications and limitations

When off-policy wins

• Real-robot data are expensive. If one hour of Scout’s time costs money, reusing each transition many times is worth the extra complexity. The SAC authors report running it directly on real robots (a quadruped and a valve-turning hand) in their 2018 applications paper.
• Data from other sources. Demonstrations (Module 3) or logs from earlier policies can sit in the buffer from the start, if the critic is not misled by them.
• Continuous control with small networks. Reparameterised gradients often have lower variance than score-function gradients.

When on-policy wins, and when off-policy fails

• Cheap simulation. With thousands of parallel simulated robots, samples are nearly free and PPO’s stability and simplicity count for more than reuse. This is why PPO dominates large-scale simulated training (Module 9).
• Sensitivity. Off-policy methods have more things that can go wrong: the buffer size, the update-to-data ratio, τ, the reward scale (which α interacts with), and critics that quietly diverge. PPO fails more loudly.
• Offline data. With a fixed dataset and no new interaction, the actor drives toward actions the data never contains, where the critic is an unchecked guess, and the minimum of two critics is not enough (Fujimoto et al. studied this extrapolation error in a separate 2019 paper). Module 8 builds the remedies (lessons M8-T3 and M8-T4).
• Non-stationary rewards or tasks. Old transitions describe a world that no longer exists.

Checklist when a SAC run misbehaves

1. Is the critic loss growing, or are the Q values far above anything the return can be? Overestimation, or a bug in the done flag (lesson M4-T6).
2. Is the entropy (or α) collapsing early or exploding? Check the target entropy and the reward scale.
3. Is the buffer large enough and recent enough, and is the first batch of data random rather than from an untrained policy?
4. Is a time-limit truncation treated as a real terminal state? It should not be (lesson M6-T3).
5. Does it fail on several seeds, or on one? Lesson M6-T7 shows how large seed-to-seed differences are, even for PPO.

For the labs of this module, none of this is run: lab M6-L3 (tune a PPO run) and lab M6-L4 (diagnose a diverging training run) use the PPO trainer. When you want to try SAC, a library implementation with its published defaults is a better start than writing your own.

Common misconceptions

Engineering takeaways

1. A replay buffer lets the learner reuse every transition, which is allowed for action-values because only the first action is fixed by the data; the price is stale data and a moving bootstrap target.
2. SAC maximises return plus α times entropy, using the soft backup y = r + γ(min of two target critics − α log π).
3. The minimum of two critics cut a +1.03 overestimate to +0.27 in a simulation, without removing the bias.
4. A tanh-squashed Gaussian needs the correction −Σ log(1 − tanh² u) in its log-probability; the entropy of a Gaussian depends on log σ (1.45 nats at σ = 0.5 against −1.77 at σ = 0.1 in two dimensions).
5. Off-policy wins when real samples are scarce and fails on fixed offline data (Module 8); this module’s trainer runs PPO, A2C and REINFORCE, and SAC here is explained, not run.

References and further reading

The two Haarnoja et al. papers define SAC; Fujimoto et al. introduced the clipped double Q-learning used in it; Mnih et al. is the replay and target-network precedent.

• T. Haarnoja, A. Zhou, P. Abbeel and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
• T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel and S. Levine, “Soft Actor-Critic Algorithms and Applications”, arXiv:1812.05905, 2018.
• S. Fujimoto, H. van Hoof and D. Meger, “Addressing Function Approximation Error in Actor-Critic Methods”, Proceedings of the 35th International Conference on Machine Learning (ICML), 2018 (TD3).
• V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature 518, 529–533, 2015.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (chapter 11 on the deadly triad).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: PPO and SAC both fail in recognisable ways. Training stability and diagnosis reads the numbers a run logs, and shows with real runs of the trainer what each failure looks like.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T5: Proximal Policy Optimization

  Write the clipped surrogate objective and explain what clipping prevents.