M6-T3 — Baselines, Advantages and GAE

Learning objectives

After this lesson, you should be able to:

• Prove that subtracting a state-dependent baseline leaves the policy gradient unbiased, and say what makes a baseline biased.
• Define the advantage A = Q − V and the TD error, and compute GAE(λ) by hand for terminated and truncated episodes.
• Explain how λ and γ trade bias against variance, with measured numbers.
• Use advantage normalisation and the critic’s explained variance sensibly, and state the terminal-versus-truncation rule.

The intuition

Scout is on a route where every step is worth about the same. Suppose a batch of steps is followed by returns of 7.2, 7.9 and 8.1. A raw policy gradient says: all three actions were followed by large returns, make all of them more likely. But the state was one where the return was going to be about 8 whatever Scout did. The informative question is whether each action did better or worse than expected from that state. The expectation is the value function V(s); the difference G − V(s) is the advantage.

The advantage can be estimated in many ways that differ in how many real rewards they add before trusting the critic. Look at one four-step episode that ends at the goal. At the last step there is nothing left to add, so all estimates agree: 1.95. At the first step they disagree sharply: 0.45 if the critic is trusted after one step, 3.12 if the whole sampled return is used. The knob λ moves between the two.

The mathematics

A baseline does not bias the gradient

Start from the policy gradient of lesson M6-T2 and replace the return G_t by G_t − b(s_t), where b may depend on the state but not on the action taken in it. The extra term is

So the expected gradient is unchanged: the baseline only changes the variance. The proof breaks if b depends on the action (b(s, a) no longer factors out of the sum), or on later rewards of the same sample, or is computed from the same batch of samples being differentiated. Lesson M6-T2 showed the last case: a baseline equal to the batch’s own mean reward gave a mean gradient of 1.197 instead of 1.6, a factor (N − 1)/N for N = 4.

The best state-dependent baseline for variance is close to, but not exactly, V^π(s) (it weights each state by the squared length of its score). V is nevertheless the standard choice: cheap to learn, interpretable, and it makes the weight an advantage.

Advantage, Q and the TD error

The TD error is the one-step advantage estimate, built from the sampled reward and the critic. It has low variance (one reward) and is biased whenever V is wrong. The Monte Carlo return minus V, G_t − V(s_t), is unbiased for the advantage (up to the baseline argument above, whatever V is) and has the variance of a whole episode of sampled rewards. Lesson M4-T5 made the same contrast for value estimation.

Generalised advantage estimation

Schulman and co-authors define the estimator as an exponentially weighted sum of the TD errors:

γ

discount (lesson M4-T2). It defines the objective being optimised, and it also shortens the horizon over which credit is assigned.

λ

the GAE parameter in [0, 1]. It does not change the objective, only how an estimate is built: how long the estimate trusts sampled rewards before it trusts V.

δ_t

the TD error at step t, using the critic at t and t + 1.

V(s_t+1) at the end of an episode

zero if the episode terminated; the critic’s value if it was truncated (below).

Terminated versus truncated

An episode can end for two different reasons. It terminates when the task is over: Scout reaches the goal or crashes into the pillar. Nothing is earned afterwards, the value of the next state is 0, and the last step’s target is the reward alone. It is truncated when the clock runs out (80 steps in this task): the world was not over, Scout could have earned more, and the next state has a value. The correct target bootstraps from the critic: δ_last = r + γ V(s_next) − V(s). Treating truncation as termination tells the critic that every state reached at step 80 is worthless; treating termination as truncation lets the value of the goal leak. Lesson M4-T6 met the same rule for Q-learning. This trainer follows it: the bootstrap is 0 when the last step reached the goal or crashed, and V(s₈₀) otherwise. A caveat on truncation: the critic does not see the time left, so its value of a state is the same at step 5 and at step 79; the bootstrap is an approximation, and a time-aware critic is a refinement.

Two knobs, two kinds of bias

Both γ and λ trade bias for variance. A γ below 1 is a different objective: with γ = 0.9 a goal 28 steps away is weighted by 0.9²⁸ = 0.052, so a +10 bonus counts as +0.52 at the start, whatever the critic. A λ below 1 does not change the objective but introduces bias when the critic is inaccurate. Schulman et al. make this distinction, and for most work γ is set near 1 (here 0.99, weighted 0.75 at 28 steps) and λ tuned (here 0.95).

Advantage normalisation and explained variance

Normalising makes the update size independent of the reward scale, so a learning rate that works for rewards of order 1 also works for order 100. It also centres the batch, so about half the steps are pushed down: relative credit is kept, absolute credit is lost. EV measures how much of the return the critic explains: 1 is perfect, 0 is no better than predicting the mean, negative is worse than that. A critic with EV near 0 or below makes the advantages mostly noise (lesson M6-T7).

A worked example

All numbers were computed in code; the first three tables can be checked with a calculator.

1. GAE by hand for a terminated and a truncated episode. γ = 0.9, λ = 0.5, so γλ = 0.45. Episode A reaches the goal on its fourth step (reward 9.95 = 10 − 0.05, terminated), episode B is cut off after four steps with a bootstrap value V(s₄) = 3.0.

Backwards with Â_t = δ_t + 0.45 Â_t+1. Episode A: Â₃ = 1.95; Â₂ = 0.65 + 0.45 × 1.95 = 1.5275; Â₁ = 0.80 + 0.45 × 1.5275 = 1.4874; Â₀ = 0.45 + 0.45 × 1.4874 = 1.1193. Episode B: Â₃ = 0.15, Â₂ = 0.2675, Â₁ = 0.3704, Â₀ = 0.4667.

At λ = 1, episode A’s advantages are G − V: the returns to go are 7.1181, 7.9645, 8.9050, 9.9500, less the critic’s 4.0, 5.0, 6.5, 8.0. The last step is 1.95 for every λ, because it has no later step.

2. The two bookkeeping mistakes. If episode B is wrongly treated as terminated, δ₃ = −0.05 + 0 − 2.5 = −2.55 instead of +0.15. At λ = 0.5 its advantages become 0.2206, −0.1764, −0.9475, −2.55 (three of four negative), and at λ = 1 all four are negative (−1.17 to −2.55): a perfectly healthy run that merely ran out of time is told that every action in it was bad. In the other direction, if the terminated episode A is given a leaked bootstrap value of 8.0 at its end, δ₃ = 9.95 + 0.9 × 8.0 − 8.0 = 9.15 and the λ = 0.5 advantages become 1.7754, 2.9454, 4.7675, 9.15: the goal is credited with its reward and then with the value of a state after the goal.

3. Normalising the batch. The eight advantages of the λ = 0.5 row, from both episodes, have mean 0.9173 and (population) standard deviation 0.6440. Normalised: 0.314, 0.885, 0.947, 1.604 for episode A and −0.700, −0.849, −1.009, −1.192 for episode B. All four steps of the truncated episode become negative, although each action was better than the critic expected: normalisation keeps only the ranking within the batch. It is a feature when the batch is large and mixed, and a trap when a batch is very small or uniformly good. The trainer normalises every batch this way for all three algorithms.

4. A λ sweep on real batches. From a PPO run (seed 1) stopped after 25 iterations, 24 positions were taken on rollouts of the stochastic policy, with two different sampled actions at each. For each action 150 continuations of the policy were simulated, GAE(γ = 0.99, λ) was computed with the run’s own critic, and the estimates were compared with a Monte Carlo advantage from 600 rollouts per next state. The table reports (a) the RMS bias of the difference between the two actions’ estimates (differences remove the state offset, which is a baseline error and does not bias the gradient) and (b) the standard deviation of one action’s estimate over the 150 continuations. Because the world is deterministic given the action, λ = 0 depends on no random continuation.

The bias falls as λ grows (0.084 to 0.023) and the spread grows (by a factor of about 40 between λ = 0.5 and λ = 1 for seed 4). Honest limits: the true differences in the first column are small (mean size 0.020, Monte Carlo noise 0.006), so only that column’s bias is resolved; for the other two the Monte Carlo noise in the reference (0.13 to 0.15) is as large as the differences, and only the standard deviations are reported. It is two checkpoints and 24 position pairs, a demonstration of the direction rather than a benchmark. The trainer’s λ = 0.95 gives a standard deviation between 0.18 and 0.9 in these three columns, well below λ = 1 in the seed 4 and 60-iteration columns.

5. How far credit reaches with γ. A 7 m trip at the maximum 0.25 m per step takes at least 28 steps. The goal bonus is weighted by γ²⁸: 0.052 at γ = 0.9, 0.238 at 0.95, 0.755 at 0.99, 0.972 at 0.999. The effective horizon 1/(1 − γ) is 10, 20, 100 and 1,000 steps. At γ = 0.9, a +10 goal 28 steps away is worth +0.52 at the start, comparable to ten step costs.

6. Explained variance of the critic in the trainer. Ten seeds, 60 iterations × 12 episodes, the value logged each iteration (EV of the critic against the batch’s λ-returns, after that iteration’s update; mean ± standard deviation over seeds):

The critic starts worse than a constant and takes tens of iterations to explain more than half of the return; the A2C critic (one step per batch) is worse than a constant for most of the run. An EV below 0 means the advantages in that iteration were built on a critic that hurts more than it helps. These are measured on the data being fitted, which flatters the critic; a held-out EV would be lower.

Engineering implications and limitations

Order of operations for a batch

1. Collect complete or truncated episodes; record, for every step, the reward, V(s), whether it terminated, and V(s′) for the last step of a truncated one.
2. Compute δ and run the GAE recursion backwards per episode, never across an episode boundary.
3. Form the critic targets (advantage + V, the λ-return) and the policy weights (advantage, normalised over the batch).
4. Update the actor with the normalised advantages and the critic towards the λ-returns.
5. Log the explained variance of the critic with the same batch.

Lab M6-L1 (compute-returns-and-advantages) is exactly this, with batches of concatenated episodes: S1 returns with resets at episode ends, S2 GAE with the terminated/truncated distinction, S3 normalisation and explained variance. The bugs it plants (return leaking across an episode boundary, a bootstrap that is not zeroed at a terminal step, λ applied to the wrong term, a constant batch) are the ones this lesson describes.

Choosing γ and λ

• Start from γ = 0.99 and λ = 0.95: the defaults of this trainer, and a common starting point in the literature.
• If the critic is bad early (EV below 0), lower λ spreads less critic error but starts biased; if returns are very noisy and the critic is good, raise λ.
• Pick γ from the time scale of the task: 1/(1 − γ) should be a little longer than the shortest route to a reward. Do not use γ to fix a reward you dislike (lesson M5-T1).

Diagnose with the right numbers

Lab M6-L3 (tune-a-ppo-run) shows EV in its plots, and lab M6-L4 (diagnose-a-diverging-training-run) has an incident whose cause is a critic that does not learn: EV near 0 means the advantages are mostly noise. Lesson M6-T4 trains the critic and the actor together, and lesson M6-T5 limits the actor’s step.

Check these before trusting a run

• Is the time-limit truncation handled with a bootstrap, and the real terminal with zero?
• Do advantages cross episode boundaries anywhere (a single backward pass over a concatenated batch)?
• Is the baseline fixed with respect to the current sample (a critic, a running mean from earlier batches), not computed from it?
• Is the EV above 0 after the first few iterations?

Common misconceptions

Engineering takeaways

1. A baseline that depends only on the state leaves the policy gradient unbiased (E[∇log π · b(s)] = 0) and reduces variance; a baseline built from the same batch or the action is biased.
2. The advantage A = Q − V says how much better an action was than usual, and the TD error δ = r + γV(s′) − V(s) is its one-step estimate.
3. GAE(λ): Â_t = Σ (γλ)^l δ_t+l = δ_t + γλ Â_t+1. λ = 0 gives the TD error (critic bias, no sampling noise), λ = 1 gives G − V (no critic bias, most noise).
4. Terminated (goal, crash): bootstrap with 0. Truncated (80-step limit): bootstrap with V(s′). Getting this wrong turned a healthy episode’s advantages negative.
5. Normalise advantages over the batch for scale-free updates, and log the critic’s explained variance: values near or below 0 mean the advantages are mostly noise.

References and further reading

Sutton and Barto give the policy gradient with a baseline; Schulman and co-authors introduced GAE and the two-knob view of γ and λ.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (chapter 13, section on REINFORCE with baseline).
• J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation”, International Conference on Learning Representations (ICLR), 2016.
• R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning 8(3–4), 229–256, 1992.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: with a learned critic supplying V and the advantages, the actor and critic train side by side in actor-critic methods (lesson M6-T4).

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T2: Policy Gradients and REINFORCE

  Derive the policy gradient estimator and write REINFORCE for a discrete and a Gaussian policy.

• Lab
  M6-L1: Compute Returns and Advantages

  The advantage estimates are wrong at episode boundaries: write discounted returns and GAE that handle terminal states.