LM12-T3 — Evaluating and Presenting Your System

Learning objectives

After this lesson, you should be able to:

• Use a held-out test set once and add an out-of-distribution slice.
• Report results with Wilson intervals, per slice, and with a categorised list of errors.
• Compare two systems with an exact paired sign test and interpret the p-value.
• Evaluate at the application level and prepare an honest demonstration.

The idea: evidence a sceptic can check

Imagine a sceptical reader with the fullest access to your repository. Everything you say should survive their checks: the test set was not used to choose anything, the intervals are computed correctly, the comparison uses a valid test, and the failures that you did not fix are listed with examples. Designing the evaluation with that reader in mind is the surest way to an honest report.

1. Held-out data, used once

Build the test split in milestone 1, before training, and look at it only for the final numbers. Add a separate out-of-distribution (OOD) slice that tests what you claim beyond memorisation: in the capstone, bay numbers never seen in training. In-distribution accuracy measures how well the model reproduces patterns it saw; the OOD slice measures generalisation, and the two can differ a lot (0.959 versus 0.820 in the capstone run).

2. Intervals

Every proportion gets a Wilson interval (Lesson LM11-T5). With 97 test examples and 93 correct the interval is 0.899 to 0.984; with 89 OOD examples and 73 correct it is 0.728 to 0.886. These are intervals for the test set sampling; the variation across seeds (Lesson LM12-T2) is separate and must be reported too.

3. Slices

An overall number averages over easy and hard inputs. Break results down by input type (here by tool), by length, by value ranges, by any property relevant to your users. Small slices have wide intervals: two correct of three emergency-stop examples is compatible with anything from 21% to 94%. Say so rather than claiming either success or failure from a handful of examples.

4. Error taxonomy and failure cases

Classify every error: here invalid JSON, wrong tool, wrong arguments. In the capstone run all 16 OOD errors were wrong arguments (the model typed a nearby bay number), none were invalid JSON; that points to digit handling for unseen numbers, not to a format problem. Include three to five concrete failing inputs with the output and the expected result: the reader learns more from them than from the table.

5. Paired comparison

If two systems were evaluated on the same examples, compare them per example. Count the examples only one gets right (a_only, b_only) and apply the exact sign test (Lab LM12-L4 Stage 3). In the capstone, masked and unmasked models disagreed on 15 OOD examples (9 against 6) and the two-sided p-value is 0.607: no evidence of a difference with this seed, even though the pooled intervals across two seeds each looked separated in milestone 2. Paired and unpaired analyses answer different questions; a report that shows both is more convincing than one that picks the more flattering.

6. Application level

The model is one component. Measure the outcome of the whole path (Module 11): how many requests are executed, how many of those are right, how many are refused and why. In the capstone the application executed 72 of 97 test commands, 71 of them exactly right, and refused 25: 22 of them correct commands that exceeded the policy’s speed limit of 0.6, by design. The safety layer lowered task completion and raised safety; both facts belong in the report.

Never tune on the test set

The mathematics: Wilson intervals and the exact sign test

Wilson interval for k successes in n trials, p̂ = k/n, z = 1.96:

Exact sign test. Let a_only and b_only count the examples on which exactly one system is right, n = a_only + b_only and k = min(a_only, b_only). Under the null hypothesis that the systems are equally good, each disagreement favours either system with probability 1/2, so

For 9 against 6: n = 15, k = 6, p = 0.607. For 8 against 2: 0.109. For 15 against 3: 0.0075. The test uses only the disagreements: examples that both systems get right or both get wrong carry no information about which is better. Its power therefore depends on the number of disagreements, and a comparison with few disagreements (two systems that are almost identical) cannot find a difference however long the test set.

Reading a p-value honestly

A p-value of 0.607 does not show that the systems are equal; it shows that this data cannot distinguish them. A p-value of 0.0075 shows that the observed split is unlikely under equality, not that the winner is better everywhere, nor that the effect is important. Report the counts together with the p-value so that the reader sees the effect size, and do not run many tests and report the smallest p (with 20 comparisons one false “significant” result is expected).

Slice reporting

Slices of a test set are dependent on the overall result and can be small. Compute an interval for each, mark the ones with fewer than about 30 examples as indicative only, and, if you looked at many slices, mention that some will look bad by chance.

A worked example: intervals, a sign test and a slice table

The code reproduces the numbers above: the Wilson intervals of the capstone test and OOD results, three sign-test p-values, and a slice table. The slice counts are an illustrative split of the 97 test examples that adds up to the real total of 93 correct; they show the shape of the report, not measurements.

import math

def wilson(k, n, z=1.96):
    p = k / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return round(centre - half, 3), round(centre + half, 3)

def sign_test(aonly, bonly):
    n = aonly + bonly
    if n == 0:
        return 1.0
    k = min(aonly, bonly)
    total, comb = 0, 1
    for i in range(k + 1):
        total += comb
        comb = comb * (n - i) // (i + 1)
    return min(1.0, 2 * total / 2 ** n)

print(wilson(93, 97), wilson(73, 89))                # (0.899, 0.984) (0.728, 0.886)
print(round(sign_test(9, 6), 4), round(sign_test(8, 2), 4), round(sign_test(15, 3), 4))    # 0.6072 0.1094 0.0075

# Per-slice reporting: a good overall number can hide a bad slice.
slices = {"move_to": (88, 90), "emergency_stop": (2, 3), "get_battery": (2, 2), "say": (1, 2)}
total_k = sum(k for k, n in slices.values()); total_n = sum(n for k, n in slices.values())
print(total_k, total_n, round(total_k / total_n, 3), wilson(total_k, total_n))          # 93 97 0.959 (0.899, 0.984)
for name, (k, n) in slices.items():
    print(f"{name:15s} {k}/{n}  {wilson(k, n)}")
# move_to         88/90  (0.923, 0.994)
# emergency_stop  2/3  (0.208, 0.939)
# get_battery     2/2  (0.342, 1.0)
# say             1/2  (0.095, 0.905)

def error_budget(counts):
    n = sum(counts.values())
    return {k: round(v / n, 3) for k, v in sorted(counts.items(), key=lambda kv: -kv[1])}

print(error_budget({"correct": 73, "wrong_args": 16}))      # {'correct': 0.82, 'wrong_args': 0.18}

Read the slice table as a reviewer would: the large slice is estimated precisely, the small ones are compatible with almost anything, and the overall interval is dominated by the large slice. Writing “the model handles all four tools” on the basis of the small slices would be an overclaim.

A demonstration that does not mislead

1. Show typical inputs,Choose them by a rule fixed beforehand (for example, the first five test examples), not because they work.
2. Show a failure,At least one real failure from your error list, with what the application did about it (refused, asked, fell back).
3. Show the safety layer,A hostile or malformed input and the reason it was refused (Module 11).
4. State what the demo does not show,Unseen phrasing, other languages, speed, real hardware.
5. Provide the script,The demo is a script in the repository; anyone can rerun it.

Engineering implications and limitations

Freeze before you test. Save the model, the configuration and the evaluation code, then run the test once and archive the output. Changes after that produce a new version and a new report.

Compute all numbers by script. The report’s tables come from a script that reads the saved predictions. Copy-paste of numbers is where discrepancies arise.

Store predictions, not only scores. With the predictions you can recompute any metric, slice, interval or paired test later, and reviewers can audit them.

Use the same examples for paired comparisons. Evaluate every system on the identical examples in identical order, and keep the per-example results.

Separate the three headline numbers. Model accuracy, application success and safety refusals answer different questions (what the model can do, what the system delivers, what the guard prevents). Do not merge them.

Look at the errors yourself. Read at least twenty failures. Patterns you find (a digit confused, a phrase unseen) guide the next experiment and the limitations section.

For robots. A demonstration on a simulator or a point on a grid shows software behaviour, not physical performance. Name the setting explicitly; do not let a software demo imply hardware validation.

Common misconceptions

Engineering takeaways

1. Use the held-out test once; add an out-of-distribution slice to measure generalisation (0.959 in-distribution versus 0.820 OOD in the capstone run).
2. Report Wilson intervals for every proportion (93/97: 0.899 to 0.984; 73/89: 0.728 to 0.886), slices with their sizes, and an error taxonomy with concrete failures.
3. Compare systems per example with the exact sign test: 9 against 6 gives p = 0.607; 15 against 3 gives 0.0075. Report counts with p.
4. Evaluate at the application level: the capstone application executed 72 of 97 commands (71 exactly right) and refused 25, 22 of them correct but too fast.
5. Freeze before testing, script every number, store predictions and show failures in the demonstration.

References and further reading

The headline numbers come from the capstone notebook LM12-L3 (CPU run of the reference solution); the slice table is illustrative, as stated. Module 9 of this track derives the interval and test formulas used here.

• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158).
• Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7).
• Liang, P. et al. (2022). Holistic evaluation of language models. arXiv:2211.09110.
• Build a Large Language Model (From Scratch), the whole book (primary book reference)
• Publisher page for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: writing the report, stating limitations and planning next steps.

Related content

Nothing below is required to finish this lesson.

• Before
  LM12-T2: Experiment Design and Reproducibility

  Plan baselines, ablations, seeds and logging so results can be reproduced.

• Lab
  LM12-L3: Capstone Milestone 3: Evaluation and Application

  Evaluate reproducibly and build a small task-specific application.