M6.4 — Training and Validation

Learning objectives

After this lesson, you should be able to:

• State the roles of the training, validation and test sets.
• Read learning curves and identify the point for early stopping.
• Choose a hyperparameter using validation error.
• Explain how reusing a test set creates optimistic results.
• Explain why validation loss is not the same as robot task success.
• Compute and read a confidence interval for a success rate.

The one-sentence idea

Fit the model on one set, make every decision on a second, and report the result on a third that was never touched.

Whatever you tune on is no longer a fair test.

Three datasets with three jobs

M6.1 showed that error on the data a model was trained on says little about how it will do on new data. The standard remedy is to split the available data into parts and give each a clear job:

Training set

The data the model’s parameters are fitted to. The gradients come from here.

Validation set

Held out during training and used to make decisions: when to stop, which learning rate or model size to use. It is looked at repeatedly.

Test set

Held out until the end, used once to report how well the final model does. It must not influence any choice.

For robot data the split is made by episode (M6.3), so that no near-duplicate of a training step is hiding in validation or test data. Typical proportions are roughly 80% / 10% / 10%, though what matters more is that each held-out set is large and varied enough to be meaningful, and that it represents the situations the robot will actually meet.

import numpy as np

rng = np.random.default_rng(0)
episode_ids = np.arange(100)                      # 100 episodes
rng.shuffle(episode_ids)
train, val, test = episode_ids[:80], episode_ids[80:90], episode_ids[90:]
print(len(train), len(val), len(test))
print(len(set(train) & set(val)), len(set(train) & set(test)), len(set(val) & set(test)))   # overlap between sets

80 10 10
0 0 0

The three sets partition the 100 episodes, with no episode in more than one (the overlaps are 0).

Learning curves and early stopping

While training, you track the loss on both the training and the validation sets after each pass, giving two learning curves. The training loss normally keeps falling. The validation loss falls at first, while the model learns real structure, and then begins to rise as it starts fitting noise. The gap opening between them is the sign of overfitting from M6.1.

The toy run below fits a degree-14 polynomial (15 parameters) to 20 noisy points by gradient descent, starting from zero, and reports both losses at several points in training:

from numpy.polynomial import legendre

rng = np.random.default_rng(1)
f = lambda x: np.sin(2 * np.pi * x)
x_tr, x_va = np.linspace(-1, 1, 20), np.linspace(-0.97, 0.97, 100)
y_tr = f(x_tr) + rng.normal(0, 0.4, 20)
y_va = f(x_va) + rng.normal(0, 0.4, 100)
Phi_tr, Phi_va = legendre.legvander(x_tr, 14), legendre.legvander(x_va, 14)     # polynomial features

w = np.zeros(15)
curve = []
for step in range(1, 100001):
    w -= 0.1 * Phi_tr.T @ (Phi_tr @ w - y_tr) / len(x_tr)                   # one gradient-descent step
    if step in (1, 10, 100, 1000, 10000, 100000):
        curve.append((step, round(float(np.mean((Phi_tr @ w - y_tr) ** 2)), 3),
                            round(float(np.mean((Phi_va @ w - y_va) ** 2)), 3)))
for row in curve:
    print(*row)

1 0.489 0.62
10 0.387 0.525
100 0.083 0.238
1000 0.02 0.193
10000 0.01 0.525
100000 0.01 0.614

The columns are the step, the training loss and the validation loss. The training loss falls all the way down (0.489 to 0.01). The validation loss reaches its lowest point (0.193) around step 1,000 and then rises, to 0.614 by the end, as the model uses its spare parameters to fit the noise in the 20 training points. Early stopping acts on this: keep checking the validation loss, and stop training, or keep the saved model, at the point where it was lowest. This is why checkpoints are saved regularly during long training runs.

Choosing settings with the validation set

Many choices are not learned by gradient descent: the learning rate, the model size, how strongly to regularise. These are hyperparameters, and the validation set is how you choose them. Here the same degree-14 model is fitted in one step by ridge regression, which adds a penalty λ ‖w‖² that discourages large weights, and the penalty strength λ is picked by validation error:

def ridge(Phi, y, lam):
    return np.linalg.solve(Phi.T @ Phi + lam * np.eye(Phi.shape[1]), Phi.T @ y)

best = None
for lam in (0.0, 0.001, 0.01, 0.1, 1.0, 10.0):
    w = ridge(Phi_tr, y_tr, lam)
    val_err = float(np.mean((Phi_va @ w - y_va) ** 2))
    print(lam, round(val_err, 3))
    if best is None or val_err < best[1]:
        best = (lam, val_err)
print("chosen lambda:", best[0])

0.0 0.614
0.001 0.595
0.01 0.471
0.1 0.22
1.0 0.221
10.0 0.467
chosen lambda: 0.1

With no regularisation (λ = 0) the validation error is 0.614, the same overfit as at the end of the run above. Moderate values bring it down to about 0.22, and a penalty that is too strong (λ = 10) underfits and the error climbs back to 0.467. The best value, 0.1, was chosen using the validation set, which brings us to the main discipline of this lesson.

Why the test set must stay untouched

Every time you pick the better of two options by looking at a held-out set, you leak a little information from it into your choices, and its error becomes optimistic. Try enough options and you will find one that happens to look good on that particular data. The effect is easy to see with models that have no skill at all. Take 100 classifiers that guess randomly, and pick the one with the best accuracy on a small test set:

rng = np.random.default_rng(5)
labels_test = rng.integers(0, 2, 20)                          # a small test set of 20 yes/no labels
labels_new = rng.integers(0, 2, 2000)                         # fresh data from the same source

guesses = rng.integers(0, 2, (100, 20))                       # 100 "models" that guess at random
acc = (guesses == labels_test).mean(axis=1)
winner = acc.argmax()
print(acc.max())                                              # the winner's test accuracy
print(round(float(rng.integers(0, 2, 2000).__eq__(labels_new).mean()), 2))     # a random guesser on fresh data

0.8
0.5

The best of the 100 coin-flippers scores 80% on the test set, which looks like real skill, while a random guesser scores 50% on fresh data. The winner was picked because it was lucky on that test set. The cure is the discipline of the three-way split: all choices are made on the validation set, and the test set is evaluated once, at the end, on the one model you chose.

For robots: validation loss is not success

Two cautions apply when the model is a robot policy:

• The loss is a proxy. Validation loss measures how well the policy predicts demonstrated actions on held-out episodes. As M6.2 showed, a policy can predict them well and still fail when its own errors compound. The final judgement is task success measured by running the robot, and the two can disagree. Phase 8 treats evaluation in detail.
• Real trials are few, so success rates are uncertain. Every trial takes time, so a success rate often rests on 10 or 20 runs. The uncertainty of a rate k/n can be expressed with a confidence interval, such as the Wilson interval:

def wilson(k, n, z=1.96):
    p = k / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return round(float(centre - half), 2), round(float(centre + half), 2)

print(wilson(7, 10))
print(wilson(70, 100))

(0.4, 0.89)
(0.6, 0.78)

Seven successes in 10 trials is a 70% rate, but the plausible range runs from about 40% to 89%, wide enough that two policies scoring 7/10 and 5/10 cannot honestly be told apart. The same 70% rate over 100 trials narrows the range to roughly 60% to 78%. Comparing policies reliably therefore needs many trials, the same conditions, and ideally repeated runs, a theme Phase 8 returns to.

Common misconceptions

Engineering takeaways

1. Training fits parameters, validation makes decisions, and the test set is for a final check.
2. Split robot data by episode into disjoint sets.
3. Training loss usually keeps falling, and validation loss turns up when overfitting starts.
4. Early stopping keeps the model from the lowest validation loss.
5. Hyperparameters such as regularisation strength are chosen on validation data.
6. Choosing among many options on a held-out set inflates its score, as 100 random guessers show.
7. Validation loss is a proxy, and robot task success has to be measured by executing the policy.
8. Small numbers of trials give wide confidence intervals.

Hold something back, and never fool yourself with it.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You can train and judge a model honestly. Next is the vocabulary of policies, and how observations, states and actions fit together in a learned controller.

Back to the VLA roadmap prerequisites