LM9-T6 — Responsible Evaluation and Model Limitations

Learning objectives

After this lesson, you should be able to:

• Explain why a handful of impressive examples is not evidence, and how selective reporting arises.
• Plan failure-focused tests: worst cases, slices, adversarial and safety probes.
• Write down the contents of an honest evaluation report, including limitations and what was not tested.
• Check a report for missing elements, and state when human oversight is required.

The idea: look for the failure, report what you find

Language models are good at producing impressive samples. A person trying a model on a handful of prompts they chose will see mostly successes, remember the best, and form an impression much better than the model’s average. This is not dishonesty; it is how selection works. Responsible evaluation replaces impressions with a protocol.

How evaluations mislead

• Cherry-picking: showing the best examples, the best seed, the best prompt wording (Lab LM9-L2 Stage 1).
• Optional stopping and multiple comparisons: trying variants until one wins (Lesson LM9-T2).
• Test-set leakage and contamination (Lesson LM7-T5).
• Averages that hide slices: a good overall score with a failing group (Lesson LM6-T5).
• Wrong metric: optimising perplexity or accuracy when users need something else.
• Distribution shift: a test set from yesterday’s users, a deployment for tomorrow’s.
• Lack of uncertainty: differences within the noise presented as findings.

Failure-focused testing

Spend as much effort looking for failures as measuring averages. Read many outputs, especially the failed ones. Build slices for the cases that matter most or are most likely to go wrong: rare inputs, long inputs, other languages, ambiguous requests, adversarial text (injected instructions, forged markers), requests that should be refused, and inputs concerning groups of people who may be treated differently. Ask red-team questions: how could someone misuse this? How could it fail silently? What is the worst thing it could output in this setting?

People and context

Evaluation does not end at the metric. Who is affected if the model is wrong, and can they notice and correct it? A suggestion tool whose output is reviewed by an expert tolerates errors that an autonomous action does not. Where an error could injure someone, damage property or cause an unfair decision, a person must review before the action, and the evaluation must test that the review step works (do reviewers actually catch the errors?).

Data, privacy and provenance

Test and fine-tuning data may contain personal information or content with restricted licences. Record where data came from, what consent and licence apply, and what was removed. Models can reproduce training text; test for leakage of sensitive strings. Keep evaluation data secure.

Say what you did not test

A limitation section is not an apology; it is the part of the report that protects the next person. List conditions you did not cover (languages, hardware, input lengths, adversaries), the assumptions of your tests (the test set is from last quarter), and the known failures with their rates.

The mathematics: how little a few examples say

Few examples. If a system fails on a fraction f of requests, the chance that n independently chosen test requests show no failure is

For f = 5% and n = 10: 0.95¹⁰ = 60%. A demo of ten successes happens six times in ten for a system that fails one request in twenty. To be 95% sure of seeing at least one failure when f = 1%, you need n ≥ ln 0.05 / ln 0.99 = 298.1, so 299 requests.

Rule of three. If you see no failures in n requests, the 95% upper bound on the failure rate is about 3/n: 30% for 10 requests, 3% for 100, 0.3% for 1,000. Claiming a failure rate below 1% needs more than 300 clean requests.

Selection. If you choose the best of k outputs per prompt, the reported quality is that of the best of k, not of the system: with a per-output success rate of 60% and k = 5, at least one of five succeeds with probability 1 − 0.4⁵ = 99%.

Subgroups. A slice of n_s items gives a margin of about 1.96 √(p(1 − p)/n_s); with 40 items and p = 0.8 that is ±12 points. You cannot detect a ten-point gap between groups with slices that small, and absence of evidence is not evidence of absence: report the size and the margin.

Review as part of the system. If a model makes an error with probability e and a human reviewer catches a fraction c of errors, the residual error is e(1 − c). With e = 5% and c = 0.9, residual 0.5%. But reviewers who see mostly correct outputs catch fewer (automation bias); measure c in the evaluation, with seeded errors, rather than assume it.

A worked example: how little ten examples say, and a report checker

The code computes the probabilities above, then checks a structured evaluation report for the elements of an honest one: sizes, intervals, slices, failure examples, limitations and versions.

import math

print(round(0.95 ** 10, 2))                                   # 0.6    ten clean examples for a 5% failure rate
print(math.ceil(math.log(0.05) / math.log(0.99)))             # 299    requests to see a 1% failure with 95% probability
print([round(3 / n, 3) for n in (10, 100, 1000)])             # [0.3, 0.03, 0.003]   rule of three
print(round(1 - 0.4 ** 5, 2))                                 # 0.99   best of five outputs at 60% each
print(round(0.05 * (1 - 0.9), 3))                             # 0.005  residual error with a 90% reviewer

REQUIRED = ["intended_use", "out_of_scope", "versions", "test_set", "results", "failure_examples", "limitations", "not_tested", "reviewed_by"]

def check_report(report):
    problems = [f"missing section: {k}" for k in REQUIRED if not report.get(k)]
    for r in report.get("results", []):
        if "n" not in r or "interval" not in r:
            problems.append(f"result '{r.get('metric', '?')}' lacks n or an interval")
        elif r["n"] < 30:
            problems.append(f"result '{r['metric']}' rests on only {r['n']} items")
    return problems

good = {
    "intended_use": "turn operator requests into one of 12 robot commands", "out_of_scope": "free-form chat; any action without operator confirmation",
    "versions": {"base": "sha256:ab12", "adapter": "sha256:cd34", "data": "v3", "code": "git:1f2e3d4"},
    "test_set": {"source": "operator logs, Q3", "n": 480, "slices": {"normal": 300, "edge": 120, "adversarial": 40, "refusal": 20}},
    "results": [{"metric": "valid command rate", "value": 0.96, "n": 480, "interval": [0.94, 0.98]}, {"metric": "refusal rate (refusal slice)", "value": 0.95, "n": 20, "interval": [0.75, 1.0]}],
    "failure_examples": ["'go left of the second shelf' -> unknown shelf id", "typo in command name -> wrong command"],
    "limitations": ["tested on text only", "operators from one site"], "not_tested": ["other languages", "noisy speech transcription"], "reviewed_by": "J. Doe, 2025-01-15",
}
print(check_report(good))                                     # ["result 'refusal rate (refusal slice)' rests on only 20 items"]
bad = {"intended_use": "assistant", "results": [{"metric": "accuracy", "value": 0.97, "n": 10}]}
for p in check_report(bad):
    print(p)

Even the good report is flagged: its refusal slice has only 20 items, too few for a number (interval 0.75 to 1.0), which the report itself shows, and which a reader should weigh. The checker cannot tell whether the report is true, only whether the questions a reader will ask are answered. The second report fails it on every count: no versions, no test set description, no failure examples, no limitations, and a headline accuracy resting on ten items with no interval.

Questions to ask of any evaluation

Engineering implications and limitations

Make the report part of the release. No model goes to users without a report that passes the checker and has a named reviewer.

Keep the failure log public inside the team. Share the worst failures, not just the average. Teams that only share successes lose their sense of the risk.

Re-evaluate on change. A new base, adapter, prompt or tool can break behaviours that used to work. Run the regression suite on every change.

Monitor in use. Log inputs and outputs (respecting privacy), sample them for review, and track the rates of refusals, invalid outputs and user corrections. Treat a drift as an incident.

Plan for the failure. Decide in advance what the system does when the model is uncertain, unavailable or wrong: refuse, ask, fall back to rules, hand over to a person.

Be careful with demos. If you show examples, say how they were chosen, and show a failure beside them.

Involve the people affected. Operators and users find failures you did not think of. Include their reports in the test set.

For robots. Tie the language-model evaluation to the robot’s safety case: which model outputs can reach actuators, which checks stand between them, and how each was tested; keep a record of the model version in every mission log.

Common misconceptions

Engineering takeaways

1. A few impressive examples are weak evidence: (1 − f)^n of seeing no failure (60% for f = 5%, n = 10); the rule of three bounds a failure rate by 3/n.
2. Look for failures: read outputs, test slices, adversarial and safety cases, and report rates, examples and the worst case.
3. Report intervals, per-slice sizes, versions, the test set, limitations and what was not tested; have it reviewed.
4. Selective reporting (best seed, best prompt, best of k outputs) inflates quality; best of five at 60% each shows 99%.
5. Design for oversight and failure: measure reviewer catch rates, fix fallback behaviour, monitor in use, re-run tests on every change.

References and further reading

The elements of the report in this lesson follow the idea of model cards and datasheets; the checker is a teaching example, not a standard.

• Mitchell, M. et al. (2019). Model Cards for Model Reporting. arXiv:1810.03993.
• Gebru, T. et al. (2021). Datasheets for Datasets. Communications of the ACM 64(12).
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

That completes Module 9. Next, Module 10 looks at the architecture and inference techniques of modern models.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T5: Inference Latency and Memory

  Estimate time to first token, tokens per second and memory for a given model and hardware.

• Lab
  LM9-L2: Build an Evaluation Harness

  Write a repeatable evaluation harness with criteria, a pass rate and failure listing, and find the prompt-sensitive cases.

• Lab
  LM9-L3: Evaluate a Tiny GPT End to End

  Compare checkpoints and decoding settings on a held-out set, with seeds, confidence intervals and a latency and memory measurement.