LM9-T2 — Task Evaluation and Test Sets

Learning objectives

After this lesson, you should be able to:

• Derive metrics and failure costs from an application description.
• Build a test set with task types and slices, separate development and final test sets.
• Compute the number of items needed for a target margin of error.
• List the habits that invalidate an evaluation (peeking, tuning on the test set, shifting the metric).

The idea: the test set is a specification

A test set says what you promise the model can do. Build it from the application. List the kinds of requests the system will receive: common ones, rare ones, hard ones, malicious ones. For each, say what a good response is and what a bad one costs. A wrong command that moves a robot is worse than a refusal; a missing hazard report is worse than a false alarm. That gives you the metrics and the weights.

Sources of items

• Real requests (logs, operator notes, support tickets), with consent and privacy handling. The best match to reality.
• Written by experts for coverage of rare and risky cases.
• Synthetic (generated by templates or by another model), cheap and large, useful for coverage and regression tests, risky as the only source since they inherit the generator’s narrowness.
• Public benchmarks, good for comparison with others, possibly contaminated and often off-task (Lesson LM7-T5).

Slices

Tag each item with the properties that might matter: task type, length, language, source, difficulty, safety category. Report the metric per slice as well as overall (Lesson LM6-T5). A good test set has enough items in every slice that matters.

Splits

Keep three sets apart. The training set is for fitting. The development (validation) set is for choosing prompts, settings and checkpoints; it is used again and again, so it becomes mildly overfitted. The final test set is used once for the number you report. Any decision taken after looking at the final test set (change the prompt, drop a bad item, choose a different metric) turns it into a development set. If you have to peek, build a new final set.

Metrics

• Exact match / pass rate against criteria for tasks with checkable answers (commands, labels, extraction, arithmetic).
• Constraint checks for format and rules: valid structure, allowed actions, length, units.
• Reference-based similarity (n-gram overlap, embedding similarity) for text with a reference, to be treated as a rough guide.
• Human or model judgement with a rubric for open-ended quality, validated against humans on a sample.
• Outcome metrics in a simulator or on the real system: did the robot reach the goal, violate a constraint?

The mathematics: how many items, how to split

To estimate an accuracy p to within a margin m (half-width of the 95% interval), you need about

m

desired half-width of the 95% interval (for example 0.03 for ±3 points).

p

the expected accuracy; if unknown use 0.5, the worst case, which gives n ≈ 0.96/m².

n

number of independent test items needed.

For p = 0.8: ±5 points needs 246 items, ±3 points needs 683, ±2 points needs 1,537. For p = 0.9 and ±2 points: about 865. Per slice the same formula applies: a slice with 30 items has a margin of about ±14 points at p = 0.8, too coarse for a decision. If you must compare two systems, the margin of the difference matters; a paired design (the same items for both) needs fewer items (Lab LM9-L1).

Stratified splitting. To keep each slice represented in the development and test sets, split within each slice: a slice with n_s items gives round(f · n_s) items to the test set at test fraction f, with at least a minimum count per slice. Rare slices may need to be oversampled or collected separately.

Decision rule. Write the rule before running: “accept if the pass rate is at least 0.95 and the lower end of the 95% interval is at least 0.90, with zero failures in the safety slice.” A rule fixed in advance cannot be bent by the result.

Multiple comparisons. If you test k variants and report the best, the chance that at least one beats a baseline by luck at the 5% level is 1 − 0.95^k: 23% for k = 5, 40% for k = 10. Correct for it, or confirm the winner on fresh data.

A worked example: sample sizes and a stratified split

The code computes the item counts above, the margin of a small slice, the chance of a lucky winner among several variants, and splits a tagged set into development and test sets within each slice.

import math
import random

def items_needed(margin, p=0.5):
    return math.ceil((1.96 / margin) ** 2 * p * (1 - p))

print(items_needed(0.05, 0.8), items_needed(0.03, 0.8), items_needed(0.02, 0.8))   # 246 683 1537
print(items_needed(0.02, 0.9), items_needed(0.05))                                   # 865 385

def margin(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)
print(round(margin(0.8, 30), 3), round(margin(0.8, 683), 3))                         # 0.143 0.03

for k in (1, 5, 10):
    print(k, round(1 - 0.95 ** k, 2))                                                # 0.05 0.23 0.4

def stratified_split(items, test_fraction, min_test=2, seed=0):
    rng = random.Random(seed)
    by_slice = {}
    for it in items:
        by_slice.setdefault(it["slice"], []).append(it)
    dev, test = [], []
    for name in sorted(by_slice):
        group = by_slice[name][:]
        rng.shuffle(group)
        ntest = max(min_test, round(test_fraction * len(group)))
        test += group[:ntest]
        dev += group[ntest:]
    return dev, test

items = [{"id": i, "slice": s} for i, s in enumerate(["normal"] * 60 + ["edge"] * 25 + ["adversarial"] * 10 + ["refusal"] * 5)]
dev, test = stratified_split(items, 0.4)
count = lambda xs: {s: sum(1 for x in xs if x["slice"] == s) for s in ("normal", "edge", "adversarial", "refusal")}
print(count(dev), count(test))    # {'normal': 36, 'edge': 15, 'adversarial': 6, 'refusal': 3} {'normal': 24, 'edge': 10, 'adversarial': 4, 'refusal': 2}

The last line shows that the rare slices end up with only 2 to 4 test items even at a 40% test fraction. That is far too few for a number: for such slices, collect more items rather than trust a percentage.

A test plan as a table

Engineering implications and limitations

Write the test plan first. Slices, sizes, metrics and the decision rule, then collect the data. It stops the evaluation from being fitted to the model.

Lock the final set. Store it read-only, with a hash, and record every time it is used. If it has been used for decisions, retire it and build a new one.

Keep a regression suite. Every failure found in production becomes a test item (in the development set), so that it cannot silently return.

Review items as carefully as code. Wrong references and ambiguous items make good models look bad. Have a second person check a sample of references.

Do not tune the prompt on the test set. Prompt engineering is training; do it on development data (Module 11).

Automate and version. One command that runs the harness, writes the report and records the model, data and code versions. Manual evaluations are not reproducible.

Include the cost of errors. If a false positive costs 1 and a false negative costs 20, report the expected cost, not only the accuracy.

For robots. Combine text-level checks (valid command, allowed action) with outcome checks in simulation and on hardware tests, and always keep a safety slice with zero tolerance for specified failures.

Common misconceptions

Engineering takeaways

1. Derive metrics and failure costs from the application; list task types and slices; build the test set from real requests where possible.
2. Keep training, development and final test sets apart; use the final test once; any decision made after peeking turns it into development data.
3. Items needed for a margin m: n ≈ (1.96/m)² p(1 − p): 246, 683 and 1,537 for ±5, ±3 and ±2 points at 80%.
4. Fix the metric and the decision rule before running; correct for trying many variants (1 − 0.95^k).
5. Stratify the splits, give rare slices enough items, add production failures to a regression suite.

References and further reading

See Chapter 7 of the book (evaluating the finetuned LLM). The sample-size formula is the standard normal approximation for a proportion; the test plan is an illustration, not a standard.

• Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding. arXiv:2009.03300 (an example of a broad benchmark and its construction).
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: sampling variability and reproducibility: how much a result moves from run to run, and how to report it.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T1: Perplexity and Its Limits

  Compute perplexity and explain what it does and does not say about usefulness.

• Lab
  LM9-L2: Build an Evaluation Harness

  Write a repeatable evaluation harness with criteria, a pass rate and failure listing, and find the prompt-sensitive cases.