LM6-T5 — Generalization and Failure Analysis

Learning objectives

After this lesson, you should be able to:

• Categorise classification errors and recognise label noise and shortcut features.
• Compute metrics by slice and explain why the overall number can hide failures.
• Estimate the uncertainty of a metric with a standard error or a bootstrap.
• Check the calibration of a classifier’s confidence and design a transfer test for a new data source.

The idea: ask where, and why, and what changes

After training, work through three questions in order.

1. Where does it fail?

Print the misclassified validation examples and read them. Group them: the model confuses two particular classes; it fails on one kind of input (long texts, numbers, a rare phrase); the label is wrong and the model is right; the example is genuinely ambiguous. A hundred errors read carefully are worth more than another point of accuracy. Then slice the metric: compute it separately for subsets defined by length, source, class, language, time, whatever might matter to users. A classifier with a 96% overall accuracy and 62% on one input type is not a 96% classifier for that input type.

2. Why does it fail?

Three common causes are not about the model at all. Label noise: annotators made mistakes or disagreed; the model’s “errors” on these examples are partly correct. Shortcuts: the training data allowed a feature that predicts the label for incidental reasons (a word that appears only in one class, a formatting habit, the data source); the model uses it and fails when it disappears (Lab LM6-L1 Stage 3). Data gaps: some kinds of input barely appear in training. Each calls for a different fix: relabel, rebalance and add data, or break the correlation.

3. What changes tomorrow?

The test set is a sample of the past. In use, inputs may come from a new sensor, a new customer, a different season, or a user who tries to fool the system. This is distribution shift. A transfer test evaluates on data from the new source, even a small hand-labelled sample, before trusting the model there. Monitoring in production watches the distribution of inputs and of predicted classes for drift.

Confidence

A classifier also outputs a probability. If, among all predictions made with 90% confidence, 90% are right, the model is calibrated. Fine-tuned networks tend to be over-confident: they say 99% and are right 90% of the time. Calibration matters whenever confidence drives a decision, such as sending uncertain cases to a human. Check it with a reliability table: bin predictions by confidence and compare the average confidence with the accuracy in each bin.

The mathematics: uncertainty and calibration

Uncertainty of a rate. An accuracy measured as k correct out of n, p̂ = k/n, has standard error

n

the number of test examples (or of examples in a slice).

p̂

the measured rate.

SE

standard error: the typical size of the sampling noise in p̂.

For 490 correct out of 500: p̂ = 0.98, SE = 0.0063, interval 0.968 to 0.992. For a slice of only 25 examples with 22 correct: p̂ = 0.88, SE = 0.065, interval 0.75 to 1.0. Small slices tell you very little; report their size.

Comparing two models. The difference of two accuracies measured on the same examples has its own noise; a difference of 0.5 points on 500 examples (2 or 3 examples) is not evidence of anything. A paired bootstrap (resample the examples with replacement many times, compare the models on each resample) or a McNemar test on the examples where they disagree gives an honest answer.

Calibration. Group predictions into confidence bins B₁, …, B_M. The expected calibration error is

where acc(B_m) is the accuracy and conf(B_m) the mean confidence in bin m. A perfectly calibrated model has ECE 0. Temperature scaling (dividing logits by a constant T > 1 fitted on validation data, Lesson LM5-T5) is a common one-parameter fix for over-confidence.

Label noise bound. If a fraction η of the test labels is wrong in a way independent of the model, a perfect classifier scores 1 − η. With η = 0.05 (a plausible figure for crowd-labelled text), any score above 95% partly reflects agreement with the noise, and small differences between models are not interpretable.

A worked example: slices, intervals and calibration

The code evaluates a set of 1,000 predictions that carry a slice name and a confidence. The data are generated, with a deliberately weak slice and over-confidence, so that you can see what the tools reveal.

import math
import random

random.seed(0)
rows = []
for i in range(1000):
    sl = random.choices(["short", "long", "numbers", "new_sensor"], weights=[40, 35, 18, 7])[0]
    pright = {"short": 0.99, "long": 0.97, "numbers": 0.90, "new_sensor": 0.62}[sl]
    correct = random.random() < pright
    conf = min(0.999, max(0.5, random.gauss(0.97, 0.03)))          # the model is about equally sure everywhere
    rows.append({"slice": sl, "correct": correct, "conf": conf})

def acc(rs):
    return sum(r["correct"] for r in rs) / len(rs)

print("overall", round(acc(rows), 3), "n =", len(rows))
for sl in ("short", "long", "numbers", "new_sensor"):
    part = [r for r in rows if r["slice"] == sl]
    a = acc(part)
    se = math.sqrt(a * (1 - a) / len(part))
    print(f"{sl:10s} n={len(part):3d}  accuracy {a:.3f} +- {1.96 * se:.3f}")

# expected calibration error with 5 confidence bins
bins = [(0.5, 0.9), (0.9, 0.95), (0.95, 0.98), (0.98, 0.99), (0.99, 1.0)]
ece = 0.0
for lo, hi in bins:
    b = [r for r in rows if lo <= r["conf"] < hi or (hi == 1.0 and r["conf"] == 1.0)]
    if b:
        ece += len(b) / len(rows) * abs(acc(b) - sum(r["conf"] for r in b) / len(b))
print("ECE", round(ece, 3))

# a bootstrap interval for the overall accuracy
random.seed(1)
boot = sorted(acc([random.choice(rows) for _ in rows]) for _ in range(500))
print("bootstrap 95% interval", round(boot[12], 3), round(boot[487], 3))

The slice table shows the weak slice (new sensor) with a wide interval because it has few examples, while the overall number is dominated by the easy slices. The ECE is positive because the model reports about 97% confidence everywhere while its accuracy on some slices is lower: it is over-confident exactly where it should be doubtful. The exact numbers depend on the generator but the pattern is robust: overall high, one slice low, confidence flat across slices.

A checklist before shipping

Engineering implications and limitations

Keep an error log. Save the misclassified examples with the model version, the true and predicted label, the confidence and your note on the cause. The log drives data collection and shows progress that a single number hides.

Define slices in advance. Choose them from the application (sources, lengths, languages, user groups, rare classes), not from where the model happens to fail, or you will find spurious weak spots among many slices. If you test many slices, expect some to look bad by chance.

Break shortcuts deliberately. Create test examples where the suspected shortcut is absent or reversed (a message with the trigger word but the other label). If the score collapses, the model relied on it.

Measure refusal. A classifier that can say “unsure” and defer to a human can be much more useful than one that always answers. Pick the confidence threshold on validation data to reach a target error rate on the answered part, and report coverage.

Monitor after release. Log inputs, predictions and confidence; track the class distribution and the share of low-confidence predictions over time; sample some for labelling. A drift in these numbers is the early warning that the world has changed.

Do not train on the errors you analysed without a new test set. If you fix mistakes you found in the validation set and then evaluate on the same set, it is no longer a clean measure.

For robots. A classifier that routes commands or triages logs sits in a safety chain. Specify which error is worse, test on data from the actual robots and conditions, and add a rule-based or human check for the cases where the model is unsure or outside its tested range.

Common misconceptions

Engineering takeaways

1. Read the errors, group them, and compute metrics per slice; the overall number can hide a failing slice.
2. Label noise bounds the measurable score, and shortcut features give high training and validation scores that disappear in use.
3. Standard error of an accuracy: √(p̂(1 − p̂)/n), about ±1.2 points at 98% on 500 examples; small slices have wide intervals.
4. Calibration: compare confidence with accuracy in bins (ECE); fine-tuned models tend to be over-confident; temperature scaling helps.
5. Test on data from the new source before trusting a model there, and monitor drift after release.

References and further reading

See Chapter 6 of the book (evaluating the model on the test set and using the model on new text) for the baseline evaluation this lesson extends.

• Geirhos, R. et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 665–673.
• Guo, C., Pleiss, G., Sun, Y. and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. arXiv:1706.04599.
• Build a Large Language Model (From Scratch), Chapter 6: Finetuning for Classification (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes Module 6. Next, Module 7 teaches the model to follow instructions.

Related content

Nothing below is required to finish this lesson.

• Before
  LM6-T4: Dataset Preparation, Labels and Metrics

  Prepare labelled data, balance classes and choose accuracy, precision, recall and F1 correctly.

• Lab
  LM6-L1: Classification Metrics and Failure Cases

  Compute a confusion matrix, precision, recall and F1, and find the examples a classifier gets wrong for the wrong reason.

• Lab
  LM6-L2: Fine-Tune a Language Model for Classification

  Add a classification head to a small GPT, compare frozen and fine-tuned variants and analyse the errors.