M6-T1 — Function Approximation and Deep RL

Learning objectives

After this lesson, you should be able to:

• Count the table entries a discretised hall needs and explain why most are never visited.
• Explain generalisation across states and what a network as value function or as policy provides.
• Compute the log-probability and entropy of a Gaussian action by hand.
• Name three things that go wrong with function approximation (moving targets, correlated data, the deadly triad) and tell on-policy from off-policy learning.

The intuition

Picture Scout in the hall of Meridian General Hospital. In the ward of Module 4 there were 25 open cells, so Q-learning could keep a row of four numbers for each and visit every row many times. In the hall Scout is at some real-valued position, say (2.31, 3.47), and nobody will ever stand in exactly that spot twice. A table indexed by position would have to chop the hall into cells first. The finer the cells, the more precisely the table can describe the world, and the fewer visits each cell gets.

A network makes the opposite trade. It has a few hundred numbers, the same for every position. Training it on a visit at (2.31, 3.47) also changes what it says at (2.40, 3.50), because both pass through the same weights. That is generalisation: experience in one place informs nearby places. It is what lets learning work in a space too large to visit, and it is also the source of every new difficulty in this module, because an update meant for one state shifts the answer in others.

The mathematics

Counting a table

The hall is 10 m by 6 m. With square cells of side h there are (10/h) × (6/h) positions; a table of Q(s, a) for a grid of 5 × 5 = 25 discrete velocity commands has 25 entries per position. The pillar occupies some cells, which are never visited. These counts are exact arithmetic (the free-cell counts come from the pillar’s collision radius of 0.95 m):

Two further multipliers are not in the table. If the pillar could be anywhere, the state would include its position, and with 0.5 m cells that is 240 × 240 = 57,600 combinations of robot and pillar positions before any action is counted. And the action is continuous: a table of Q(s, a) needs a max over a to act and to bootstrap, and for a continuous a there is no table row to take a max over.

A function in place of a table

Replace the table by a function with parameters θ that maps a state (or a state and action) to a number:

value function:   V(s; θ) ≈ V^π(s)           (a "critic")
action value:     Q(s, a; θ) ≈ Q^π(s, a)
policy:           π(a | s; θ)             (an "actor")

squared-error fit of a critic to targets yt:
  L(θ) = mean over t of ( yt − V(st; θ) )²

The network of lesson M2-T4 is the function here. In this module’s trainer the actor is 4 inputs → 32 tanh units → 2 outputs (4 × 32 + 32 + 32 × 2 + 2 = 226 parameters) and the critic is 4 → 32 → 1 (193 parameters). The observation is the same four numbers as in Module 1, [dx, dy, px, py], divided by (5, 3, 5, 3) so that all inputs are of order one (lesson M2-T2).

A Gaussian policy for continuous actions

The actor outputs the mean μ(s) of a Gaussian over the two velocity components, and a learned log standard deviation log σ for each (one per action dimension, not a function of s in this trainer). Acting means sampling:

μ_k(s)

the network output: the velocity command Scout would send in state s without exploration noise.

σ_k

the standard deviation of exploration noise in component k. Large σ explores more and acts less precisely.

log π(a | s)

the log of the probability density of the action taken. A density can exceed 1, so log π can be positive.

H

differential entropy. It can be negative: σ = 0.1 gives −1.77 nats for two dimensions. It falls as the policy becomes more deterministic (lesson M6-T7 reads it as a diagnostic).

The environment clips each component to [−1, 1] (lesson M1-T2 introduced the action range). The trainer samples from the unclipped Gaussian, scores the unclipped action, and lets the world do the clipping. That is a simplification; an alternative is to squash the sample through tanh and correct the density, a choice common in off-policy methods such as the one in lesson M6-T6.

What changes and what can go wrong

With a table, a Bellman update (lesson M4-T3) changed one entry and nothing else. With a network, three new difficulties appear.

• Moving targets. A critic is trained towards a target such as r + γ V(s′; θ) that contains the critic itself, and the policy that defines V^π is changing too. The thing being fitted keeps moving.
• Correlated data. Successive steps of an episode are neighbours in state space, and a batch from one policy is not an independent sample of the world. Gradient descent assumes samples that look like draws from a fixed distribution (lesson M2-T3).
• The deadly triad. Sutton and Barto (2018, chapter 11) call the combination of function approximation, bootstrapping (targets that use the current estimate) and off-policy learning (data from a different policy than the one being evaluated) the deadly triad: with all three present the value estimates can diverge, and examples show it with simple linear functions. With any two, practice is much safer. Monte Carlo targets remove bootstrapping, on-policy data removes the off-policy part, and a table removes the approximation.

On-policy and off-policy at a glance

A worked example

All numbers below were computed with RLCore or by direct arithmetic; run counts are stated with each.

1. A Gaussian log-probability by hand. The actor says μ = (0.3, −0.2), both σ = 0.5. Scout samples a = (0.5, −0.6). Then z = ((0.5 − 0.3) / 0.5, (−0.6 + 0.2) / 0.5) = (0.4, −0.8).

The density is e^−0.8516 = 0.427. The entropy with σ = 0.5 is 2 × (log 0.5 + ½ log(2πe)) = 1.4516 nats, which is exactly the value the trainer logs at the start (1.4517 in the first iteration). If training squeezed σ to 0.25 the entropy would be 0.065; at 0.1 it would be −1.767; at 0.05, −3.154.

2. The tabular count against what is visited. A table of 0.25 m cells has 960 positions. Take an untrained actor with σ = 0.5 and run 60 batches of 12 episodes (the default 720 episodes of the trainer): 57,490 transitions. They touch 324 of the 960 cells (34 %) and 87 of the 240 half-metre cells (36 %). The 24,000-entry table would have most of its entries at their initial value after 57,000 steps of experience. The 226-number network is updated by every one of the transitions.

3. A moving target. Estimate V(s₀) at s₀ = (2, 3.5) by averaging the discounted return (γ = 0.99) of 400 rollouts from s₀ with each policy. Untrained actor: −2.76. The same architecture after 60 iterations of PPO from seed 1 (the run of the next example): +5.17. A critic trained during that run is chasing a target that moves by about 8 return units while it learns, and every policy update moves it again.

4. Correlated data. Along the expert path from (2, 3.5) (31 steps) the first observation component dx has lag-1 autocorrelation 0.90, and consecutive observations differ by at most 0.25 m per axis. Ten successive steps carry far less than ten steps of information about the hall.

5. The Module 6 trainer on this task. RLCore.trainPolicyGradient trains the Gaussian actor and a critic on exactly the Module 1 reach task (80 steps, reward −0.05 per step, +10 at the goal, −5 and an end on touching the pillar) with algo set to 'reinforce', 'a2c' or 'ppo'. Defaults: 60 iterations of 12 episodes, γ = 0.99, GAE λ = 0.95, learning rate 3e-3 with Adam, initial σ = 0.5. For each iteration it logs the mean return, success rate, entropy, approximate KL, clipped fraction, critic loss, explained variance and mean σ. Ten seeds of PPO, 60 iterations × 12 episodes each, then the mean policy (no noise) evaluated on 50 fresh starts:

Seven seeds learn the task and three do not find the goal within the budget. A single seed would have said either 0.92 or 0.00. That spread is why this module reports several seeds (lesson M6-T7, and lesson M1-T5 on evaluation). The logged σ hardly moves in 60 iterations (0.4999 for seed 1) because the trainer updates log σ with a tenth of the network’s step size: in these short runs the exploration noise is effectively fixed at its initial value, and the learning signal is in the mean.

What the simulation simplifies. One obstacle, no heading or inertia, noise-free sensing and a 2-D action. Real continuous control has contact, delay and high-dimensional actions, and the trainer is a teaching implementation (full-batch updates, no parallel environments), not a tuned library.

Engineering implications and limitations

Scale the inputs, then check the outputs

Networks are sensitive to input scale; a position in metres and a heading in radians differ by an order of magnitude. Normalise observations (lesson M2-T2) and keep the same normalisation at training and deployment (lesson M1-T4). Many implementations also initialise the final layer of an actor with small weights so that the first actions are small; this teaching trainer uses a standard uniform initialisation instead, and its untrained mean command is already about (−0.16, −0.33) at position (2, 3) for seed 1.

Choose the action representation

• Continuous commands such as velocities and torques fit a Gaussian actor naturally (lesson M6-T2).
• A small set of discrete skills fits a softmax actor or a Q-network with a max.
• A distribution with one peak cannot represent two equally good ways round the pillar; the mean of left and right is straight into it (lesson M3-T6). A mixture or a different head is needed.

What to log from the first run

Because the targets move and the data correlate, the training loss is not enough. Log the return, the success rate, the entropy, the critic’s explained variance and the KL between successive policies from the start (lesson M6-T7, lab M6-L4). They are cheap, and a run that fails will usually show the cause in one of them.

Check your understanding with the labs

Lab M6-L1 (compute-returns-and-advantages) puts hands on the quantities the critic is trained towards. Lab M6-L3 (tune-a-ppo-run) lets you change the learning rate and batch size of this trainer and read the diagnostics.

1. State the task, the observation scaling and the action bounds in the experiment log.
2. Run at least three seeds before believing a curve.
3. Evaluate the deterministic mean policy on held-out starts as well as the training return.
4. Keep one run with an expert or a simple baseline (lesson M1-T5) for comparison.

Common misconceptions

Engineering takeaways

1. A table needs one entry per state and action: 1,500 to 150,000 for Scout’s hall from 1 m to 0.1 m cells, and a continuous action has no row to take a max over.
2. A network shares its parameters across states (226 numbers for the actor), so experience generalises; the price is that updates interfere.
3. A Gaussian policy outputs a mean and has a std: log π = Σ[−½ z² − log σ − ½ log 2π], entropy = Σ[log σ + ½ log 2πe]. Worked example: log π = −0.8516, entropy 1.4516.
4. Moving targets, correlated data and the deadly triad (approximation + bootstrapping + off-policy) are the new failure sources.
5. The Module 6 trainer learns Scout’s reach task from scratch, but not for every seed: PPO succeeded on 7 of 10 in the default budget.

References and further reading

Sutton and Barto cover approximation, the deadly triad and policy approximation; the deep Q-network paper is the best-known application of a network as a value function.

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018 (chapter 9: on-policy prediction with approximation; chapter 11: off-policy methods with approximation and the deadly triad; chapter 13: policy gradient methods).
• V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature 518, 529–533, 2015.
• J. N. Tsitsiklis and B. Van Roy, “An analysis of temporal-difference learning with function approximation”, IEEE Transactions on Automatic Control 42(5), 674–690, 1997.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the actor does not have to be trained through a value function at all. Lesson M6-T2 derives the policy gradient and writes REINFORCE for the Gaussian actor.

Related content

Nothing below is required to finish this lesson.