M2-T5 — Generalization and Overfitting

Learning objectives

After this lesson, you should be able to:

• Use training, validation and test sets for their separate jobs, and say why the test set is touched once.
• Read a degree sweep or a training curve and name underfitting, a good fit and overfitting.
• Describe bias and variance in words, and apply ridge regularisation and early stopping to reduce variance.
• Read a learning curve to decide whether more data, a bigger model or a simpler one is the next step.

The intuition

Imagine fitting a curve through fifteen noisy points. A straight line misses the bend: it is wrong in a consistent way whichever fifteen points you draw. A wiggly curve that threads every point is wrong in a different way: it has chased the noise, so a different fifteen points would give a very different curve. The first error is bias, the second variance. Too simple a model has high bias; too flexible a model, given too little data, has high variance.

You cannot see either from the training error alone, because the wiggly curve has the lowest training error. You see them by scoring on points the model did not fit. When training error is low and held-out error is high, the model has memorised; when both are high, it cannot yet represent the pattern.

The mathematics

Three data sets, three jobs

Every time you look at the validation error and change something, you leak a little information about the validation set into the model. After many choices the validation error is optimistic. The test set exists to catch this, and it only works if nothing was chosen by looking at it. How to cut these sets from robot logs without leaking is lesson M2-T6.

The error decomposition

For a new input x, a model fitted to a random training set has expected squared error:

Capacity trades one against the other: a bigger model lowers bias and raises variance. More data lowers variance without raising bias. The sweep above is the picture: degree 1 is all bias, degree 9 on fifteen points is all variance.

Regularisation: ridge and weight decay

Instead of limiting the model’s size, limit the size of its weights. Ridge regression adds a penalty to the squared error:

The bias term is normally not penalised. λ = 0 is the ordinary fit; larger λ pulls the weights towards zero and the curve towards smooth. For a network trained by gradient descent the same penalty appears as weight decay: each step also shrinks every weight by a factor, w ← (1 − 2ηλ) w − η ∂L/∂w. λ is chosen on the validation set, never on the training set, where the best λ is always zero.

(RLCore’s ridgeFit adds λ to the diagonal of XᵀX without the factor n, which is the same idea with λ rescaled by n; the sweep below uses that convention.)

Early stopping

A network trained by gradient descent starts from small weights and fits the smooth, large-scale part of the data first, the noise later. Evaluate the validation error as training proceeds, keep the weights from its minimum, and stop when it has not improved for a set number of checks (the patience). Early stopping limits the effective size of the weights without changing the loss, and it also saves training time.

Learning curves

Plot training and validation error against the number of training samples n. Three shapes cover most cases:

• Both curves high and converging: high bias. More data will not help; the model is too simple or the features are wrong.
• Training low, validation high, a wide gap that narrows as n grows: high variance. More data, a smaller model or regularisation helps.
• Both low and close: the model has what it needs. The remaining error is the noise floor σ².

σ²

noise variance in the targets; here 0.2² = 0.04.

λ

regularisation strength.

n

number of training samples (for robots, count episodes as well as steps; lesson M2-T6).

A worked example

Everything below was computed by running RLCore’s ridgeFit and a small network in Node. The task is the curved corridor y = 0.8 sin(2x) for x in [−2, 2] with Gaussian noise of standard deviation 0.2, so the best possible mean squared error on new noisy data is 0.04. Polynomial features are the powers of x/2 up to the chosen degree. The training set has 15 points; validation and test sets have 200 points each, drawn independently.

1. The degree sweep (ordinary least squares, λ = 0):

Degrees 1 and 2 underfit: training error 0.147 and 0.142 are far above the noise floor (a sine is odd, so the even degree-2 term adds almost nothing). Degree 3 is the choice by validation error (0.192). From degree 5 on the training error is about as low as it can go while the validation error grows by orders of magnitude and the weights grow with it. Choosing the degree by training error would have picked 8 or 9 and given a model thousands of times worse on new data (468 against 0.192). The test error at degree 3, 0.274, is higher than its validation error, 0.192: with 15 training points even the “winner” is noisy, and choosing it by validation error made the validation score slightly optimistic.

2. Ridge at degree 9 (same 15 points, penalty λ on all powers, bias unpenalised):

Raising λ shrinks the weights from a norm of 556 to 2.0 and raises the training error from 0.043 to 0.094, while the validation error falls from 468 to 0.102: close to the noise floor, and better than the best unregularised degree-3 model (0.192). Past λ = 1 the weights are squeezed so hard that the model underfits again. The best λ here (0.1) is where training and validation error meet; it is chosen on validation error, and the test error, 0.100, agrees.

3. Early stopping. A 1 → 16 → 1 tanh network (lesson M2-T4) trained by full-batch gradient descent, step size 0.05, on the same 15 points:

Training error never rises, but validation error is lowest near 30,000 steps (0.056, checked every 50 steps up to 5,000 and at the listed counts after that) and then turns up, ending at 0.292, five times worse. Keeping the weights from the best validation checkpoint gives a test error of 0.047, close to the noise floor of 0.04, and the same network trained to the end is at 0.436.

4. Learning curves (mean over 30 training sets of each size, scored on the 200 validation points; degrees 1 and 3 unregularised, degree 9 with λ = 1e−4):

The line’s two curves meet at 0.29 and stay there: more data never helps a model that cannot represent the bend. The cubic converges to about 0.06. The degree-9 model has a huge gap at n = 10 (0.012 against 5.58), which closes as n grows, and at 320 points it is at 0.051, the best of the three: capacity is only a problem when data are scarce.

How much data is enough? There is no single count. In this experiment the lightly regularised 10-parameter model was at 0.323 with 20 points and within 0.005 of its 320-point value (0.056 against 0.051) from 80 points on, about eight times its parameter count; with a well-chosen penalty (λ = 0.1) even 15 points gave 0.102. The count that matters for a robot is independent episodes or situations, not logged steps (lesson M2-T6).

What the simulation simplifies. One smooth one-dimensional function, Gaussian noise of known size and 200-point validation sets. Real validation sets are small, their noise is not Gaussian, and the “true” error is not known. A single draw of 15 points is also one sample: the extreme errors at degrees 7 to 9 depend on gaps in where the points happened to fall, and another draw would give different (usually similar in size) numbers.

Engineering implications and limitations

Hold out before you look

Set aside validation and test data at the start, and write down how they were cut. Anything fitted on the data, including normalisation statistics and feature selection, uses the training set only (lesson M2-T6).

Watch the gap, not just the number

Log training and validation error together. A validation error that is good in absolute terms but far from the training error is a warning about the next change of conditions. A validation error that is poor and close to the training error is a features-or-model problem, not a data problem.

Choose regularisation by validation

Sweep λ on a log scale (1e−4, 1e−3, ...) and take the value near the minimum of the validation curve. Use the same loop for network size, weight decay and the stopping patience. Keep the final choice, then touch the test set once.

Report test error with an interval

A 200-point test set gives a noisy estimate; report how noisy, as in lesson M1-T5. For a policy, the final test is a closed-loop success rate on new scenarios, not only a prediction error (lesson M2-T6).

When the gap will not close

Try, in order: more varied data, fewer or better features, a smaller model, stronger regularisation, early stopping, data augmentation that reflects real variation. Lab M2-L3 practises reading these signs on a model that scored 99 percent on logged data.

Common misconceptions

Engineering takeaways

1. Use training data to fit, validation to choose, and a test set once, to report.
2. Training error always falls with capacity; the validation error has a minimum (degree 3 in the sweep) and then rises.
3. Error is noise plus bias plus variance; simple models fail by bias, flexible models on little data fail by variance.
4. Ridge and weight decay shrink the weights: λ = 0.1 turned a degree-9 fit from 468 to 0.102 validation error.
5. Early stopping keeps the checkpoint with the lowest validation error; learning curves say whether more data, a bigger model or a simpler one is the cure.

References and further reading

Standard sources on the bias-variance trade-off, ridge regression and early stopping.

• T. Hastie, R. Tibshirani and J. Friedman, “The Elements of Statistical Learning”, 2nd edition, Springer, 2009 (chapters 3 and 7).
• A. E. Hoerl and R. W. Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems”, Technometrics 12(1), 1970.
• S. Geman, E. Bienenstock and R. Doursat, “Neural Networks and the Bias/Variance Dilemma”, Neural Computation 4(1), 1992.
• L. Prechelt, “Early Stopping - But When?”, in Neural Networks: Tricks of the Trade, Springer, 1998.
• I. Goodfellow, Y. Bengio and A. Courville, “Deep Learning”, MIT Press, 2016 (chapter 7, Regularization for Deep Learning).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the validation set only tells the truth if it is cut correctly. Lesson M2-T6 shows how neighbouring robot steps leak, and how to split by episode, day and robot.

Related content

Nothing below is required to finish this lesson.

• Before
  M2-T4: Neural Networks from Scratch

  Write the forward pass and the gradients of a small multilayer network, and say what each layer adds.

• Lab
  M2-L3: Catch the Overfit Model

  The model scores 99 percent on logged data and fails on the floor: find the leak and the overfit, and fix the evaluation.