LM9-T4 — Hallucination and Prompt Sensitivity

Learning objectives

After this lesson, you should be able to:

• Define hallucination in a way that can be tested, distinguishing unsupported from contradicted claims.
• Explain why next-token training produces fluent unsupported text.
• Measure prompt sensitivity with paraphrase sets and consistency metrics.
• Compute a coverage-accuracy trade-off for a model that may abstain, and list mitigations.

The idea: probable is not the same as true

A language model is trained to produce text that is likely given the preceding text (Lesson LM5-T1). Likely text is fluent and plausible. Whether it is true depends on whether the model learned the relevant fact, and a model that does not know a fact has no built-in way to say so: the most probable continuation of “The capital of Freedonia is” is a plausible-sounding name. The result is hallucination.

Definitions you can test

• Contradicted claim: a statement that conflicts with a reference (a document, a database, a known fact).
• Unsupported claim: a statement that the reference neither supports nor contradicts. It may be true or false, but the system had no basis for it. In summarisation and question answering over a document, this is the common failure.
• Fabricated entity: an invented name, citation, function or part number.

“Faithfulness” (to the provided source) and “factuality” (to the world) are different properties. Evaluate the one your application needs, against a source you can check.

Why it happens

• The training objective rewards plausibility, not truth or calibrated doubt.
• Rare facts are seen few times and are poorly encoded.
• Instruction tuning on answers the model could not have known teaches it to answer anyway (Lesson LM7-T1).
• Decoding with a high temperature or a long output amplifies small errors: one wrong token becomes context for the next.
• The prompt may invite it: a question that presupposes a false premise.

Prompt sensitivity

The same request phrased differently can give different answers: reordered options, a different template, a typo, a polite prefix. A robust system gives the same correct answer under reasonable paraphrases. Sensitivity makes single-prompt evaluations unreliable (Lab LM9-L2 Stage 1) and shows that part of the model’s behaviour is surface pattern matching.

Measuring

• Reference-grounded checks: compare answer claims with the source (string match for entities and numbers, or a verifier model validated on human labels).
• Claim-level scoring: split an answer into atomic claims and label each supported, unsupported or contradicted; report the rate of unsupported claims per answer.
• Abstention and coverage: allow “I don’t know” and report accuracy on answered items together with coverage.
• Paraphrase sets: for each test item, several rewordings; report mean accuracy, worst-case accuracy, and consistency.
• Adversarial probes: questions with false premises, non-existent entities, requests to cite sources.

The mathematics: unsupported-claim rate, coverage and consistency

If answers are split into claims and u_j of the c_j claims of answer j are unsupported, the unsupported-claim rate is

The answer-level rate A is much larger than U for long answers: with a per-claim rate of 3% and 20 claims per answer, a fraction 1 − 0.97²⁰ = 46% of answers contain at least one unsupported claim.

Selective prediction. If the model has a confidence c_i for answer i (for example the probability of the answer tokens, or a separate score) and answers only if c_i ≥ τ:

τ

the confidence threshold; raising it reduces coverage.

correct_i

1 if the answer is right, else 0.

coverage(τ)

the share of questions the system answers.

accuracy(τ)

accuracy on the answered questions only.

For 20 questions with the confidences and outcomes of the code below: answering all gives accuracy 0.45; τ = 0.5 answers 65% with accuracy 0.62; τ = 0.7 answers 45% with accuracy 0.78; τ = 0.9 answers 20% with accuracy 1.0. A useful confidence is one for which accuracy rises as τ rises (the curve in the figure); a useless one gives a flat curve. Calibration (Lesson LM6-T5) is what makes the threshold meaningful.

Consistency. For N items and k paraphrases, with outcomes o_ij ∈ {0, 1}, let an item be sensitive if 0 < Σ_j o_ij < k. Then

where acc_j is the accuracy under paraphrase j. A system with 90% mean accuracy and consistency 0.6 gets 40% of the items right under some wordings and wrong under others.

A worked example: abstention and consistency in code

The code computes the coverage–accuracy table for 20 answers with confidences, and the consistency, per-paraphrase accuracy and spread for a small outcome table.

items = [(0.99, 1), (0.97, 1), (0.95, 1), (0.9, 1), (0.88, 0), (0.85, 1), (0.8, 1), (0.75, 0), (0.7, 1), (0.65, 0),
         (0.6, 0), (0.55, 1), (0.5, 0), (0.45, 0), (0.4, 0), (0.35, 0), (0.3, 1), (0.25, 0), (0.2, 0), (0.1, 0)]   # (confidence, correct)

for tau in (0.0, 0.5, 0.7, 0.9):
    kept = [c for conf, c in items if conf >= tau]
    print(tau, round(len(kept) / len(items), 2), round(sum(kept) / len(kept), 2))
# 0.0 1.0 0.45
# 0.5 0.65 0.62
# 0.7 0.45 0.78
# 0.9 0.2 1.0

# consistency over four paraphrases of six items (rows = items, columns = wordings)
m = [[1, 1, 1, 1], [1, 1, 0, 1], [0, 0, 0, 0], [1, 0, 0, 0], [1, 1, 1, 1], [0, 1, 0, 0]]
k = len(m[0])
sensitive = [i for i, row in enumerate(m) if 0 < sum(row) < k]
per_wording = [sum(row[j] for row in m) / len(m) for j in range(k)]
print(sensitive, round(1 - len(sensitive) / len(m), 2))                     # [1, 3, 5] 0.5
print([round(a, 2) for a in per_wording], round(max(per_wording) - min(per_wording), 2))   # [0.67, 0.67, 0.33, 0.5] 0.33

# answer-level rate from a per-claim rate
print(round(1 - 0.97 ** 20, 2))                                              # 0.46

Two lessons from the output: a model that is right 45% of the time overall can be right 100% of the time on the fifth of questions it is most sure about; and a model with a 50% pass rate in the paraphrase table is right on only two items under every wording.

Mitigations

Engineering implications and limitations

Ground the evaluation in a source. For document tasks, keep the source with each test item and check claims against it. For open-ended facts, use a trusted knowledge base for the slice you care about.

Evaluate on questions the model cannot answer. Include non-existent entities and false premises, with “cannot answer” as the correct response, and report the abstention rate on them.

Use paraphrase sets. Write 3–5 rewordings per test item (or generate and review them). Report mean, worst case and consistency. Randomise the order of multiple-choice options.

Prefer scores you can audit. A verifier model is itself a model; validate it against human labels on a sample before trusting its numbers.

Show sources to the user. Provide the passages the answer was drawn from, so that a person can check. A citation the model fabricated is worse than none; verify cited sources exist and support the claim.

Do not use the model’s fluency as evidence. Confident, detailed answers are as likely to be fabricated as vague ones.

Gate high-stakes outputs. For anything that triggers an action, require a valid structure, check values against the allowed set, and route low-confidence cases to a person.

For robots. A hallucinated part number, coordinate or action name is a hazard. Constrain outputs to the known inventory, check them against the environment state, and treat “unsupported” as a reason to ask, not to act.

Common misconceptions

Engineering takeaways

1. Hallucination: unsupported or contradicted claims or fabricated entities; faithfulness (to a source) and factuality (to the world) are different.
2. It arises because training rewards plausible text; long answers multiply the chance (3% per claim over 20 claims gives 46% of answers with an error).
3. Selective prediction: coverage(τ) and accuracy(τ); a useful confidence shows accuracy rising as coverage falls (0.45 → 1.0 in the example).
4. Prompt sensitivity: evaluate paraphrase sets and report mean, worst case, consistency and spread.
5. Mitigate with retrieval, constrained output, verification and abstention, and evaluate each mitigation on questions the model cannot answer.

References and further reading

See Chapter 7 of the book for evaluating responses of an instruction-tuned model; the coverage-accuracy table and consistency numbers in this lesson are computed by the code shown.

• Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12).
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the cost of inference: latency, throughput and memory.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T3: Sampling Variability and Reproducibility

  Measure run-to-run variation, set seeds and report confidence intervals.

• Lab
  LM9-L2: Build an Evaluation Harness

  Write a repeatable evaluation harness with criteria, a pass rate and failure listing, and find the prompt-sensitive cases.