M6-T5 — Proximal Policy Optimization

Learning objectives

After this lesson, you should be able to:

• Compute the probability ratio and the clipped surrogate for samples with positive and negative advantages, inside and outside the clip range, and say which ones give a gradient.
• Explain that clipping is a heuristic trust region on samples, not a bound on the KL divergence, and cite measurements that show the KL still growing.
• Read the KL and clip-fraction diagnostics, and describe early stopping, minibatches, the entropy bonus and advantage normalisation.
• Compare PPO with TRPO at the level of the idea, and interpret a clip-by-epochs sweep over seeds without over-reading it.

The intuition

Imagine adjusting a recipe from tasting notes written on one evening. A small change in the seasoning is safe. A large change based on that one evening’s notes can ruin the dish, and you only find out the next time you cook, when the notes about the new dish are the ones you have. A policy is the same: the data in the batch describes how the old policy behaved, and tells you less and less about the new one as it moves.

PPO measures how far each action’s probability has moved with the ratio r = π_new(a | s) / π_old(a | s). With a positive advantage, raising the ratio is good, until r passes 1 + c, after which the objective stops rewarding it. With a negative advantage, lowering the ratio is good, until r passes 1 − c. Beyond those points the sample contributes no gradient. The figure shows the two cases for c = 0.2.

The mathematics

The ratio and the surrogate

The policy gradient at the current parameters can be written as the gradient of the surrogate L(θ) = E[ r_t(θ) Â_t ], where the expectation is over data from π_old. At θ = θ_old the ratio is 1 and ∇r = ∇ log π, so the surrogate’s gradient is the usual advantage-weighted policy gradient. Away from θ_old the surrogate is a valid local approximation only. Maximised without limit, it would push the ratio of every positive-advantage sample as high as it can go.

c

the clip range, typically 0.1 to 0.3. The default of the trainer here is 0.2.

min

the pessimistic choice: of the unclipped and the clipped term, take the lower. The objective is never more optimistic than r·A.

Â_t

the advantage from the critic (lessons M6-T3 and M6-T4), usually standardised over the batch.

log π_old

stored when the batch is collected. The old policy is not recomputed during the epochs.

Which samples give a gradient

For each sample the gradient of the objective with respect to log π_θ(a | s) is r·Â when the minimum selects the unclipped term and 0 when it selects the clipped constant. Working through the four sign cases gives:

Two of the four cells keep the gradient even when the ratio is outside the band: a positive-advantage action whose probability has fallen below 1 − c, and a negative-advantage action whose probability has risen above 1 + c. In both the policy has moved the wrong way, and the objective lets the update repair it.

Epochs and minibatches

One batch is used for E epochs. In each epoch the data may be split into minibatches, with one gradient step per minibatch, so the batch gets E × M steps. The first step has ratio 1 exactly and the clip is inactive; with E = 1 and a full batch the clip can never act (measured below). Later steps start from a policy that has already moved.

Diagnostics

What the clip does not guarantee

The clip acts on the ratio of sampled actions, one at a time, and removes only a gradient. It does not bound the KL divergence between the policies, nor the ratio of any particular sample. Reasons: a gradient step that improves some samples moves the shared weights and shifts others, including samples that are already outside the band; optimiser momentum (Adam’s running averages) can keep moving the weights for a while where a gradient is now zero (a likely contributor; I did not isolate it); and samples in the “wrong-way” cells keep full gradients. Engstrom et al. (2020) report that code-level optimisations, not the clip itself, account for much of PPO’s measured advantage over TRPO, and that the clip does not reliably keep the ratios inside the band. PPO is therefore a heuristic trust region. The KL is a diagnostic to watch, not a constraint you can assume.

Entropy bonus and advantage normalisation

The term −c_e H(π) rewards a wider policy, which resists premature collapse of the action standard deviation. Normalising advantages to mean 0 and standard deviation 1 per batch makes the step independent of the reward scale and balances positive and negative advantages (the mean is zero).

Compared with TRPO

Trust Region Policy Optimization (Schulman et al., 2015) maximises the same surrogate subject to an explicit bound on the mean KL divergence between the old and new policy. It solves this with a second-order natural-gradient step (using conjugate gradients) and a line search that rejects steps that violate the bound or fail to improve the surrogate. PPO (Schulman et al., 2017) replaces the constraint by the clipped objective, which needs only first-order optimisation (Adam) and several epochs on the same batch, and is much simpler to code and to combine with shared networks. The price is that the trust region is implicit; TRPO controls it directly.

A worked example

Every number was produced by running RLCore.trainPolicyGradient on Scout’s reach task (as in lesson M6-T4: 12 episodes per iteration, 60 iterations, lr 3 × 10⁻³, Adam, γ = 0.99, λ = 0.95, advantages standardised per batch, clip c = 0.2, 5 epochs unless stated). The learner is full-batch, so one epoch is one gradient step.

1. A table of real samples. The policy after 40 iterations (seed 3) is π_old; the policy after 41 iterations, trained on its own batch, is π_new. I then drew 12 new episodes (886 steps) with π_old and computed the ratio and the advantage (old critic, GAE, standardised) of each step. Because the update did not see these samples the ratios here are what a held-out batch would show, not exactly what the training batch showed. Rows are one example of each case (log-probabilities are rounded to three decimals, so r may differ from the quotient by 0.001); c = 0.2:

Check row 2: r  = 1.289 × 0.777 = 1.002; the clipped term is 1.2 × 0.777 = 0.932; the smaller is 0.932, a constant, so there is no gradient. Row 5: r  = 0.770 × (−0.188) = −0.144 and the clipped term is 0.8 × (−0.188) = −0.150; the smaller is −0.150, again a constant. In row 3 the unclipped term is the smaller (0.575 against 0.589), so the sample keeps its gradient even though r is outside the band; row 6 is the same on the other side. Over all 886 samples:

So 53 of 886 samples (5.98%) are clipped, and 111 (12.5%) are outside the band; the other 58 outside it still push. The ratios range from 0.652 to 1.731 with a median of 1.003 and 5th and 95th percentiles 0.800 and 1.234. The mean of the unclipped surrogate over the batch is −0.0013 and that of the clipped one is −0.0040. The sample estimate of the KL is mean(log π_old − log π_new) = 0.0015 by k1 and mean((r − 1) − log r) = 0.0089 by k3: the two disagree by a factor of 6 on 886 samples, which is the noise in k1 on a small batch.

2. Clip range and epochs, with measured KL. 5 seeds (1 to 5), 60 iterations, PPO. “KL” is the mean over iterations of the approximate KL(old ‖ new) after the update; “max KL” is the mean over seeds of the largest KL in the run, with the single worst seed in brackets; “clip” is the mean clip fraction in the final epoch; “eval” is the success of the mean policy on 50 new starts, averaged over seeds.

What this supports. With one epoch the clip never acts (clip fraction 0) and all four clip ranges give identical runs, as the argument above says. A larger clip range gives a larger KL: at 5 epochs the mean KL goes 0.0011, 0.0042, 0.0178, 0.0467 for c = 0.05, 0.1, 0.2, 0.4. More epochs do not reliably raise the mean KL: from 5 to 10 epochs it fell for c = 0.1 and 0.2 (0.0042 to 0.0035, 0.0178 to 0.0126), stayed about equal for 0.05 (0.0011 to 0.0012) and rose for 0.4 (0.0467 to 0.0612), because the clip silences samples as the policy moves. The worst single update still escaped: seed spikes of 0.259 (c = 0.2, 10 epochs) and 0.287 (c = 0.4, 10 epochs) show the KL escaping the band by an order of magnitude, which is the “heuristic trust region” point. Over 10 seeds the default (c = 0.2, 5 epochs) had a mean KL of 0.0162 per update and a mean maximum of 0.055.

What it does not support. The best eval success is 0.768 (c = 0.2, 5 epochs) and 0.752 (c = 0.4, 5 epochs), and 0.200 at c = 0.2 with 10 epochs. The per-seed standard deviation of success is about 0.45, so with 5 seeds the standard error of each cell is about 0.2: only the very small clips (0.05 at 5 and 10 epochs, 0.1 at 5 epochs: 0.18 to 0.22) look worse than c = 0.2 at 5 epochs, and even that is a difference of only about 2 standard errors of the difference. I cannot claim from these data that 10 epochs hurts at c = 0.2, or that 0.4 is as good as 0.2. The pattern “too little movement is slow, too much movement is unstable” is the textbook story and is consistent with the table, but five seeds do not establish it. The 60-iteration budget also matters: the small clips are probably slow, not broken.

3. Entropy bonus. PPO, 5 seeds, c_e = 0, 0.01, 0.1 and 1: final action standard deviation 0.500, 0.500, 0.504 and 0.547; eval success 0.768, 0.572, 0.400 and 0.568. In this implementation the log-standard-deviation is a bounded parameter updated with a small plain-gradient step, so the bonus barely moved the entropy, and the differences in success are seed-to-seed noise, not an effect of exploration. The bonus is real in theory (and in implementations with a faster-moving standard deviation); this experiment cannot show it helping. Advantage normalisation is always on in the trainer, so I have no ablation of it.

What the simulation simplifies. A full-batch update means one gradient step per epoch; a typical implementation splits a batch of 886 samples into minibatches of 64 (14 of them), so 5 epochs would be 70 steps, not 5, and the ratios drift much further. There is one process, no vectorised environments, 12 episodes per batch, and no KL early stopping, no value clipping and no learning-rate schedule. The absolute numbers say nothing about large robots or long tasks.

Engineering implications and limitations

Settings that usually work, and what to watch

• Start from the defaults: c about 0.2, 3 to 10 epochs per batch, Adam with a learning rate near 3 × 10⁻⁴ for larger networks (the trainer here uses 3 × 10⁻³ for a small one), advantages standardised per batch. These are conventions from the literature, not results of this lesson’s measurements.
• Watch the KL per update: values of the order of 0.01 to 0.02 are typical of healthy runs, and a KL that is repeatedly 10 times that, or a clip fraction near 1, means the steps are too large (lab M6-L4). These are rules of thumb; the numbers depend on the action space.
• Early stopping on KL: many implementations stop the epochs for a batch when the mean KL exceeds a target (often about 1.5 times a chosen value) and discard the rest. It enforces the trust region that the clip only encourages. The trainer here does not implement it, so I have no measurement of its effect.
• Minibatches: each batch is split into pieces and visited in a random order each epoch; this gives more, noisier steps, so reduce the epochs or the learning rate when you add them.
• Learning rate and clip interact: both set how far the policy moves per batch. Change one at a time and read the KL and the clip fraction after each change (lab M6-L3).
• Entropy and the action noise: a bonus of 0.0 to 0.01 is common; if the standard deviation collapses early, raise it or lower the learning rate. A learned log-standard-deviation can behave differently from a state-dependent one.

When PPO is the right tool

PPO is a robust default for simulated robot training with many parallel environments and cheap samples, such as locomotion in a physics simulator. It is on-policy, so it needs fresh data for every batch and can be wasteful on a real robot; off-policy methods (lesson M6-T6) reuse old data. It is also sensitive to implementation details, so a published result may not reproduce from the objective alone (Engstrom et al., 2020).

Debug in this order

1. Look at the critic first (explained variance, value loss; lesson M6-T4): bad advantages make the clip pointless.
2. Look at the KL and the clip fraction per iteration (lesson M6-T7). Zero clip fraction means the clip is idle; a large KL means the steps are too big.
3. Look at the entropy: a fall to near zero early stops exploration.
4. Change one hyperparameter at a time and compare means over at least five seeds, because a single seed can mislead (the standard deviation of success across seeds above is about 0.45).

Lab M6-L2 has you write the clipped objective, its gradient mask and the two KL estimates. Lab M6-L3 lets you change the clip range, epochs, learning rate and entropy coefficient on Scout and read the curves.

Common misconceptions

Engineering takeaways

1. PPO reuses a batch for several epochs and limits the damage with the ratio r = π_new / π_old and the objective min( r  , clip(r, 1 − c, 1 + c)  ).
2. A sample gives zero gradient only when it has already moved far enough in the direction its advantage favours ( > 0 and r > 1 + c, or  < 0 and r < 1 − c); the other out-of-band samples keep their gradient so mistakes are repaired.
3. The clip is a heuristic trust region. It does not bound the KL: on Scout the mean KL per update was 0.0126 at c = 0.2 with 10 epochs, with a worst spike of 0.259.
4. Watch the KL and the clip fraction. A clip fraction of 0 means the clip never acted (one epoch); a high one means steps are too large. Early stopping on KL enforces what the clip only encourages.
5. Sweeps over 5 seeds had a standard error near 0.2 in success: they reveal the KL trend with the clip range clearly and the ranking of success settings only weakly. TRPO enforces a KL bound directly with a second-order step; PPO does it approximately, with first-order steps.

References and further reading

The primary sources for PPO and TRPO, the generalised advantage estimator used for the advantages, and a study of the implementation details behind PPO’s behaviour.

• J. Schulman, S. Levine, P. Abbeel, M. Jordan and P. Moritz, “Trust Region Policy Optimization”, ICML, 2015.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford and O. Klimov, “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.
• J. Schulman, P. Moritz, S. Levine, M. Jordan and P. Abbeel, “High-Dimensional Continuous Control Using Generalized Advantage Estimation”, ICLR, 2016.
• L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph and A. Madry, “Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO”, ICLR, 2020.
• S. Kakade and J. Langford, “Approximately Optimal Approximate Reinforcement Learning”, ICML, 2002 (the surrogate and the case for small policy steps).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: lab M6-L2 has you implement the clipped objective, its gradient mask and the KL estimates, and lab M6-L3 lets you tune a run. Lesson M6-T6 turns to SAC and off-policy learning, and lesson M6-T7 collects the diagnostics that separate a healthy run from a broken one.

Related content

Nothing below is required to finish this lesson.

• Before
  M6-T4: Actor-Critic Methods

  Describe an actor and a critic, how they are trained together, and what bias they introduce.

• Lab
  M6-L2: Implement the PPO Clipped Objective

  Write the clipped surrogate loss and its gradient signal, and see which samples it silences.

• Lab
  M6-L3: Tune a PPO Run

  Change learning rate, clip range, batch size and entropy bonus for a simulated Scout and read the curves that follow.