LM6-T4 — Dataset Preparation, Labels and Metrics

Learning objectives

After this lesson, you should be able to:

• Split labelled data into training, validation and test sets with stratification.
• Compute class weights for an imbalanced task and use them in the loss.
• Compute the confusion matrix, accuracy, precision, recall, F1, and macro and weighted averages.
• Explain label agreement (Cohen’s kappa) and how label noise limits what can be measured.

The idea: data decide, metrics measure

Where labels come from. Someone, or some process, decided that each example has the label it has. That decision may be a person reading messages, a rule, an earlier model, or an outcome observed later (a robot task that failed). Each source has its own errors and biases, and the classifier will learn them along with the signal. Write down how labels were produced; two annotators who disagree 20% of the time cap what any model can achieve on that label.

Splitting

Split into training (to fit), validation (to choose settings and stop) and test (to report once). With imbalanced classes use stratified splits so that each set has about the same share of each class; with only 80 positives in 1,000 examples a random split could easily put 5 positives in the validation set and make its F1 meaningless. Split before looking at errors and before any tuning, and make sure that near-duplicates (the same message with a different timestamp) end up in the same split.

Imbalance

When one class is rare, the loss is dominated by the common class, and the model can reduce it by under-predicting the rare one. Options: class weights that multiply the loss of rare-class examples; resampling (repeat rare examples or drop common ones); a decision threshold chosen after training for the desired balance between false alarms and misses. These affect the probabilities the model reports as well; a model trained with weights is not calibrated to the true base rate.

The metrics

• Accuracy: the share of examples classified correctly. Fine for balanced classes; misleading otherwise.
• Precision: of the examples predicted positive, the share that are positive. High precision means few false alarms.
• Recall (sensitivity): of the actual positives, the share found. High recall means few misses.
• F1: the harmonic mean of precision and recall; it is low if either is low.

Which one matters depends on the cost of each error. A safety stop that must not miss hazards values recall; a message filter that deletes user mail values precision.

The mathematics: metrics, averages, weights and agreement

For a positive class, the confusion matrix has four counts: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN), with N = TP + FP + FN + TN.

TP

positives predicted positive.

FP

negatives predicted positive (false alarms).

FN

positives predicted negative (misses).

TN

negatives predicted negative.

With TP = 60, FP = 20, FN = 20, TN = 900 (N = 1,000): accuracy 0.96, precision 0.75, recall 0.75, F1 0.75. Answering “negative” always gives TP = 0, FN = 80, TN = 920: accuracy 0.92, recall 0, and precision and F1 undefined, taken as 0.

Several classes. Compute precision, recall and F1 for each class treating it as positive against the rest. Macro F1 averages them with equal weight, so rare classes count as much as common ones. Weighted F1 weights each class by its number of true examples. Micro averaging pools the counts, and for single-label classification it equals accuracy. Example with three classes and confusion matrix [[50, 5, 5], [10, 30, 0], [2, 3, 15]] (rows are true classes): the per-class F1 are 0.820, 0.769 and 0.750; macro F1 = 0.780, weighted F1 = 0.791, and accuracy = 95/120 = 0.792 (the diagonal sums to 95 of N = 120).

Class weights. A common choice weights class c by

with K classes and n_c examples of class c. For 1,000 examples with 80 positives: w_pos = 1,000/(2 · 80) = 6.25 and w_neg = 1,000/(2 · 920) = 0.543. Each positive then counts 11.5 times as much as a negative in the loss, so that both classes carry equal total weight.

Label agreement. Two annotators agree on a fraction p_o of examples. Some agreement happens by chance, p_e. Cohen’s kappa corrects for it:

If both annotators say “class A” 80% of the time, p_e = 0.8² + 0.2² = 0.68; with p_o = 0.9, κ = (0.9 − 0.68) / 0.32 = 0.69: substantial but far from perfect agreement. A classifier cannot be reliably better than the labels it is judged against.

Uncertainty of a metric. An accuracy measured on n examples has a standard error of about √(p(1 − p)/n). For p = 0.98 and n = 500 that is 0.0063, a 95% interval of about ±1.2 points. Differences smaller than that are noise.

A worked example: metrics and weighted loss in code

The first block computes every metric of the example above from the counts and from label lists. The second uses class weights in a PyTorch loss and shows how they change the loss of a rare-class error.

import torch
import torch.nn.functional as F

def scores(tp, fp, fn, tn):
    n = tp + fp + fn + tn
    prec = tp / (tp + fp) if tp + fp else 0.0
    rec = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
    return round((tp + tn) / n, 3), round(prec, 3), round(rec, 3), round(f1, 3)

print(scores(60, 20, 20, 900))        # (0.96, 0.75, 0.75, 0.75)
print(scores(0, 0, 80, 920))          # (0.92, 0.0, 0.0, 0.0)  the always-no classifier

# macro and weighted F1 from a 3-class confusion matrix (rows = true class)
cm = [[50, 5, 5], [10, 30, 0], [2, 3, 15]]
f1s, support = [], []
for k in range(3):
    tp = cm[k][k]
    fp = sum(cm[i][k] for i in range(3)) - tp
    fn = sum(cm[k]) - tp
    f1s.append(scores(tp, fp, fn, 0)[3])
    support.append(sum(cm[k]))
print(f1s, round(sum(f1s) / 3, 3), round(sum(f * s for f, s in zip(f1s, support)) / sum(support), 3))   # [0.82, 0.769, 0.75] 0.78 0.791

# balanced class weights and their effect on the loss
n, npos = 1000, 80
w = torch.tensor([n / (2 * (n - npos)), n / (2 * npos)])
print([round(v, 3) for v in w.tolist()])          # [0.543, 6.25]
logits = torch.tensor([[2.0, -2.0], [-2.0, 2.0]])   # two confident mistakes
target = torch.tensor([1, 0])                       # the first is a rare-class example (class 1), the second a common one (class 0)
plain = F.cross_entropy(logits, target, reduction="none")
weighted = F.cross_entropy(logits, target, weight=w, reduction="none")
print([round(v, 3) for v in plain.tolist()], [round(v, 3) for v in weighted.tolist()])   # [4.018, 4.018] [25.113, 2.184]

# an accuracy interval
import math
p, m = 0.98, 500
print(round(1.96 * math.sqrt(p * (1 - p) / m), 4))   # 0.0123

With class weights the same mistake costs 25.1 on a rare-class example and 2.2 on a common one, a ratio of 11.5, which is why the optimizer works harder on the rare class. With the default reduction="mean", PyTorch divides the weighted sum by the sum of the weights in the batch, so the overall scale of the loss stays comparable to the unweighted one while the balance between the classes changes.

Engineering implications and limitations

Fix the evaluation protocol before training. Decide the metric, the splits and the decision rule first. Choosing the metric after seeing results is a way of fooling yourself.

Report more than one number. Accuracy, per-class precision and recall, the confusion matrix, and the number of examples per class. Report the interval or the spread across seeds.

Do not tune on the test set. Every decision made by looking at test performance turns it into a validation set. Evaluate on it once, at the end.

Threshold after training. If the cost of errors changes (a new safety requirement), re-choose the threshold on validation data; you do not need to retrain.

Truncation and padding. Cap the length at what the model supports; truncate consistently and check how many examples are cut. Pad to the longest in the batch to save compute (Lesson LM6-T2).

Deduplicate and check leakage. Identical or near-identical examples across splits inflate scores. Hash the normalised text and keep duplicates in one split.

Audit the labels. Sample a hundred examples, relabel them yourself, and measure agreement with the existing labels. If it is below what your model scores, the metric is measuring label noise.

Keep a data card. Source, collection period, labelling instructions, known gaps. It is what lets a colleague, or you in six months, judge whether the model still applies.

Common misconceptions

Engineering takeaways

1. Split into train, validation and test with stratification, before tuning, and keep near-duplicates together.
2. accuracy = (TP + TN)/N, precision = TP/(TP + FP), recall = TP/(TP + FN), F1 = 2PR/(P + R); on 1,000 examples with 80 positives, always answering “no” has accuracy 0.92 and recall 0.
3. Macro F1 treats classes equally, weighted F1 by support; balanced class weights are N/(K·n_c).
4. Cohen’s kappa κ = (p_o − p_e)/(1 − p_e) measures annotator agreement beyond chance and bounds what a model can show.
5. A metric on n examples has noise of about √(p(1 − p)/n); small differences are not evidence.

References and further reading

See Chapter 6 of the book (preparing the dataset: balancing, splitting, and creating data loaders). The metric definitions are standard; the numbers in this lesson are worked examples that you can reproduce with the code.

• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
• Build a Large Language Model (From Scratch), Chapter 6: Finetuning for Classification (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: looking at errors and judging whether a classifier will generalise.

Related content

Nothing below is required to finish this lesson.

• Before
  LM6-T3: Frozen and Trainable Parameters

  Choose which layers to train, and compute the memory and quality trade-offs.

• Lab
  LM6-L1: Classification Metrics and Failure Cases

  Compute a confusion matrix, precision, recall and F1, and find the examples a classifier gets wrong for the wrong reason.