M6-T4 — Actor-Critic Methods

Learning objectives

After this lesson, you should be able to:

• Write the one-step TD actor-critic update and the A2C update with GAE advantages, and say which quantity scales the actor gradient.
• Explain why an imperfect critic biases the actor update, and which part of its error does not.
• Describe shared against separate actor and critic networks, the value-loss coefficient, and why a batch is used once on-policy.
• Read measured results of A2C, REINFORCE and PPO on Scout, with the spread over seeds, and say what they do and do not show.

The intuition

Think of a coach and an athlete. The athlete (the actor) tries moves. A pure return-based learner is told only the final score of the whole match and must guess which move mattered. A coach (the critic) watches every moment and says “that position was better than I expected” or “worse”. The athlete reinforces moves that led to better-than-expected positions. That is faster than waiting for the score. It has one catch: the coach is also learning, and may be wrong, and a wrong coach teaches the athlete wrong habits.

For Scout, “better than expected” is the temporal-difference (TD) error: the reward just received, plus the critic’s value of the new position, minus the critic’s value of the old one. If it is positive, the step went better than the critic predicted, so the actor makes that action more likely.

The mathematics

The two networks and their targets

The actor is a policy π_θ(a | s). On Scout it is a Gaussian: the mean is a network of the observation and the standard deviation is a learned parameter per action dimension. The critic is a state-value function V_w(s), a second network. Both are fitted from the same batch of experience.

δ_t

the TD error. It is a one-step estimate of the advantage A(s_t, a_t) = Q − V (lesson M6-T3).

α_a, α_c

the actor and critic step sizes. The same δ drives both networks.

γ

the discount (0.99 on Scout). A time-limit cut-off is bootstrapped from V, a goal or a crash is not (lab M6-L1).

A2C: batches, n-step returns and GAE

Advantage actor-critic (A2C) collects a batch of episodes with the current policy, computes an advantage Â_t for every step, and takes one gradient step on a combined loss. The advantage estimator is generalised advantage estimation (lesson M6-T3), a weighted sum of TD errors:

Â_t

the advantage estimate. In the actor term it is treated as a constant, so no gradient flows from the actor into the critic through it. Many implementations standardise it over the batch to mean 0 and standard deviation 1.

c_v

the value-loss coefficient. It only matters when both heads share parameters or share an optimiser step size (see below).

c_e

the entropy bonus (lesson M6-T5). Zero by default here.

Where the critic enters the actor update

The actor’s gradient is the REINFORCE gradient with the return replaced by the advantage: ∇J ≈ E[ ∇ log π(a | s) ·  ]. The critic therefore decides the sign and size of each step. Where  is positive the policy moves toward the sampled action, where it is negative it moves away, and where it is near zero the sample barely counts. A critic that is flat or wrong everywhere gives advantages that are mostly noise.

Shared against separate networks

Why the critic biases the actor

Suppose the critic has an error ε(s) = V_w(s) − V^π(s) (the true value of the current policy). The expected TD error then is

The term −ε(s) depends only on the state. It acts like a baseline, so it adds noise but no bias to the gradient. The term γ E[ε(s′)] depends on the next state, which depends on the action, so it does bias the gradient: the actor is pushed toward actions that lead to states the critic overestimates. With λ = 1 the advantage is a return minus V(s), the next-state term disappears and the estimate is unbiased, at the price of the full variance of the return. The choice of λ (lesson M6-T3) sets the position between the two. Compatible function approximation (Sutton et al., 1999) gives exact conditions under which the critic does not bias the gradient; practical networks do not meet them.

On-policy data: each batch is used once

The policy gradient is an expectation under the policy that collected the data. After one actor step the policy has changed and the old samples come from a different distribution. A2C therefore takes one step per batch and throws the batch away. Reusing it needs a correction for the change of policy, and that is what the ratio of lesson M6-T5 (PPO) provides. The cost of discarding data is sample efficiency, which the comparison below measures.

A worked example

Every number below was produced by running RLCore.trainPolicyGradient on Scout’s reach task (hall 10 × 6 m, goal (9, 3), pillar (5, 3), 80 steps, γ = 0.99, 12 episodes per batch, 32 hidden units, initial action standard deviation 0.5, advantages standardised per batch, separate actor and critic networks, Adam). Seeds 1 to 10 select the random stream and the start positions. “Eval” is the success rate of the deterministic mean policy on 50 new start positions per seed; “≥ 0.8” counts seeds with eval success of at least 0.8.

1. One TD error by hand. Seed 1, PPO, 60 iterations. The mean policy from one start position reached the goal in 38 steps. The critic’s value V, the one-step TD error δ = r + 0.99 V(s′) − V(s), and the Monte Carlo advantage G − V (G the discounted return actually obtained) are:

Check step 0: −0.05 + 0.99 × 2.295 − 1.745 = 0.477. Only at the last step does the one-step error see the goal reward. Along the approach (steps 34 to 36) δ is small and negative, because each step costs 0.05 and the critic’s value falls as the goal gets closer, while the actual return there is 9.5 to 9.8. The critic says about 6, the truth for this run is about 9.8: it is too low by roughly 3 near the goal. Some of this is mismatch rather than error: the critic was trained on the noisy sampling policy, which sometimes misses or crashes, and this episode is the noise-free mean policy, which reached the goal from all 12 test starts.

2. How much the critic is trusted: λ. The same critic and the same 12 noise-free episodes (385 steps), advantages with GAE at four values of λ:

At λ = 0 the estimate is built almost entirely from the critic’s own predictions, so it hides the critic’s average error (the mean advantage is near zero). At λ = 1 the real returns reveal that the critic is too low by 1.36 on average on these states. The spread roughly doubles. This is the bias-variance trade-off in numbers: the left end is smoother but trusts the critic, the right end is honest but noisier. The spread here is across the 385 samples, a proxy for noise, not a measured estimator variance.

3. A2C against REINFORCE against PPO. 60 iterations of 12 episodes (720 episodes), 10 seeds, default learning rate 3 × 10⁻³. Runtime is Node.js time per run on one core.

Per seed, the standard deviation of eval success is 0.42 to 0.47, so the standard error of a mean over 10 seeds is about 0.14. The 0.26 against 0.29 gap between A2C and REINFORCE is far inside that. PPO is better in this table, by about 0.4, which is about 2 standard errors of the difference: suggestive, not conclusive. Many seeds simply have not found the goal after 720 episodes (eval success 0.00). With more iterations:

A2C needs about 2.5 times as many episodes as PPO to reach the same level here, so for the same running time it is equally good: PPO is more sample-efficient, A2C is cheaper per iteration. REINFORCE at 150 iterations is behind A2C by 0.36, about 1.8 standard errors. The trainer’s “REINFORCE” standardises the returns-to-go over the batch, which already subtracts a crude constant baseline, so this is not the textbook baseline-free algorithm.

4. The learning rate. A2C, 60 iterations, 10 seeds, one learning rate for both networks. KL is the approximate KL(old ‖ new) after each update, averaged over iterations.

The low rate is too slow for this budget (small steps, and a critic that has not caught up yet); the high rate takes steps with KL above 2 nats in the worst update and a critic that explains none of the return variance (explained variance ≤ 0 means no better than predicting the mean). The best result is in between. The two networks share a single learning rate in this implementation, so the experiment cannot say whether the actor or the critic is the one that fails at 3 × 10⁻²; it shows only that the pair breaks down together. With PPO at the same settings, the rates 10⁻³, 3 × 10⁻³ and 10⁻² gave eval success 0.382, 0.680 and 0.558.

5. The value-loss coefficient, and a surprise. A2C, 10 seeds, c_v = 0.05, 0.5, 5 and 50 gave identical results (eval success 0.260, eval return −0.95). The reason is that this implementation uses separate networks and Adam, whose update is nearly unchanged when the gradient is multiplied by a constant. The coefficient only matters here when it is 0: the critic never trains (its explained variance stays negative, about −7 on average) and eval success is 0.096. With a shared trunk, or with plain SGD, c_v would change the effective step of the critic relative to the actor, which is why implementations with shared trunks tune it (0.5 is a common default).

6. λ in the training loop. A2C with λ = 0, 0.5, 0.95 and 1: eval success 0.184, 0.106, 0.260 and 0.266, with 2, 1, 2 and 2 seeds at or above 0.8. All are inside the noise of 10 seeds (standard error about 0.14), so these runs do not show which λ is best, and the usual advice to use λ near 0.95 comes from the literature and from example 2, not from this table.

What the simulation simplifies. One process, full-batch updates (no minibatches), no vectorised environments, 12 episodes per iteration, a single obstacle and small networks. The seeds that never find the goal are a feature of this small budget, not of the algorithms in general; larger batches and more iterations would change the ranking and the absolute numbers.

Engineering implications and limitations

Check the critic before the actor

Log the critic’s explained variance, 1 − Var(R̂ − V) / Var(R̂). A value near 1 means the critic predicts the returns; near 0 means it explains nothing, and the advantages are then mostly noise (as in the A2C run with c_v = 0). Negative values mean the critic is worse than a constant. If it is poor, fix the critic first: a larger learning rate for the critic, more critic updates per batch, better input scaling (this implementation divides the observation by fixed constants so the inputs have magnitudes near 1) or a longer return horizon. Lab M6-L1 has you compute these quantities, and lab M6-L4 shows what a critic that does not learn looks like in the curves.

Choices that interact

• Learning rates. The actor changes the data distribution, the critic must follow a moving target, and the actor depends on the critic. Too fast an actor outruns the critic; too slow a critic gives stale advantages. Many implementations use a separate, often larger, rate for the critic. Here one rate drives both.
• Batch size. A batch is an estimate of an expectation. With 12 episodes the gradient is noisy; more episodes per iteration reduce the noise at the cost of fewer updates per environment step.
• Advantage standardisation. Dividing by the batch standard deviation keeps the step size independent of the reward scale. It also hides the scale of the critic’s error, so also look at the raw advantages when debugging.
• Bootstrapping at the time limit. A cut-off after 80 steps is not a terminal state: bootstrap from V. Treating it as terminal teaches the critic that the world ends, and its value drops near the cut-off (lab M6-L1).
• Entropy. The policy’s own noise is the exploration in policy-gradient methods (compare lesson M4-T7). If the standard deviation shrinks too early, exploration stops; an entropy bonus resists that (lesson M6-T5).

On-policy against off-policy

Because A2C and PPO discard each batch after use, they are called on-policy. They are simple and stable but need many environment steps, which is costly on a real robot and cheap in a simulator. Off-policy methods (lesson M6-T6) reuse a replay buffer of old data and trade that sample efficiency for more delicate stability.

Where this leads

The next lesson (M6-T5) keeps the actor-critic structure but lets the actor take several steps on each batch, with a clip that stops the policy from drifting too far from the one that gathered the data. Lesson M6-T7 collects the diagnostics (return, entropy, KL, value loss, explained variance) that tell a healthy run from a broken one.

Common misconceptions

Engineering takeaways

1. The actor is the policy; the critic is a learned value function. The TD error or GAE advantage from the critic scales each actor step: positive makes the action likelier, negative less likely.
2. A critic reduces variance and adds bias: E[δ] = A + γE[ε(s′)] − ε(s), where only the next-state error term biases the gradient. λ moves between the TD end (smooth, trusts the critic) and the Monte Carlo end (honest, noisy).
3. A2C uses each batch once because the gradient is an expectation under the current policy. On Scout, 60 iterations gave 0.260 success for A2C and 0.680 for PPO (10 seeds); A2C reached 0.752 with 150.
4. Monitor the critic (explained variance, value loss) before the actor. At learning rate 3 × 10⁻² the A2C critic explained none of the variance and single updates moved the policy by KL 2.1.
5. Measure over seeds: the standard deviation across seeds was about 0.45 for success, so 10-seed gaps under 0.3 are not evidence. These runs are small, single-process and full-batch.

References and further reading

The textbook chapter on actor-critic methods and the primary sources for the critic, for GAE, for the asynchronous and synchronous variants, and for policy gradients with function approximation.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (chapter 13, policy-gradient methods, including actor-critic methods).
• V. R. Konda and J. N. Tsitsiklis, “Actor-Critic Algorithms”, Advances in Neural Information Processing Systems 12, 2000.
• R. S. Sutton, D. McAllester, S. Singh and Y. Mansour, “Policy Gradient Methods for Reinforcement Learning with Function Approximation”, Advances in Neural Information Processing Systems 12, 2000 (compatible function approximation).
• J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, “High-Dimensional Continuous Control Using Generalized Advantage Estimation”, ICLR, 2016.
• V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver and K. Kavukcuoglu, “Asynchronous Methods for Deep Reinforcement Learning”, ICML, 2016 (A3C, whose synchronous form is called A2C).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lesson M6-T5 turns the single-step A2C update into several clipped steps per batch (PPO) and measures how much the clip range and the number of epochs matter. Lab M6-L1 practises the advantage computation, and lab M6-L3 lets you change the settings above and read the curves.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T3: Baselines, Advantages and GAE

  Reduce variance with a baseline, define the advantage, and compute generalized advantage estimates.

• Lab
  M6-L3: Tune a PPO Run

  Change learning rate, clip range, batch size and entropy bonus for a simulated Scout and read the curves that follow.