LM11-T3 — Retrieval-Augmented Generation

Learning objectives

After this lesson, you should be able to:

• Describe the offline indexing and online query stages of a RAG system.
• Choose chunk size and overlap and count the chunks.
• Compute recall@k and mean reciprocal rank, and decompose answer accuracy into retrieval and generation.
• Assemble a grounded prompt and design for the case in which the context has no answer.

The idea: open-book answering

Closed-book answering asks the model to rely on whatever its weights remember. That fails for private, recent or rare facts, and the failures are fluent (hallucination, Module 9). Open-book answering gives the model the relevant pages with the question. The model then has an easier job: read and extract, instead of remember.

Stage 1: split documents into chunks

Models and retrievers work on pieces of limited size, and a whole manual embeds poorly: its vector is an average of many topics. Documents are therefore cut into chunks of a few hundred words, with some overlap so that a fact on a boundary appears whole in at least one chunk. Small chunks are precise but lose context and multiply the index; large chunks keep context but blur the match and use up the prompt budget (Lab LM11-L1 shows this trade-off). Respecting structure (sections, paragraphs, table rows) usually beats cutting at a fixed number of words. Keep with every chunk its source, title and position so answers can cite it.

Stage 2: retrieve

The question is turned into a vector the same way as the chunks (Lesson LM11-T2) and the top k chunks by similarity are returned. A common improvement is two steps: retrieve 20 to 50 candidates cheaply, then re-rank them with a more expensive model that reads question and passage together, and keep the best few.

Stage 3: assemble the prompt and generate

The prompt contains the instruction (answer only from the passages; say “I don’t know” otherwise; cite passage numbers), the passages in clearly delimited blocks, and the question. The delimiters and the statement that the passages are untrusted data are the first line of defence against injected instructions (Lesson LM11-T6). The model generates; the application can check that the cited passage numbers exist and that quoted text really occurs in the passages.

What RAG fixes and what it does not

• Fixes: missing or stale knowledge, private data, traceability (you can show the source), updates without retraining (re-index the changed documents).
• Does not fix: the model ignoring or misreading the passage, retrieval returning the wrong passage, questions that need combining many passages, or the prompt budget. It adds failure modes of its own: stale indexes, hostile documents, leaked access rights.

Fine-tuning versus retrieval

The mathematics: chunk counts, metrics and error composition

Number of chunks

A text of n words cut into chunks of s words with overlap o advances by s − o words per chunk:

For n = 1,000, s = 200, o = 50: ⌈800/150⌉ + 1 = 7 chunks; with no overlap 5; for a 150-word text 1. Overlap increases the index size by the factor s/(s − o) (here 1.33).

Retrieval metrics

For each question the gold passage is the one that contains the answer. Let rank_q be its position in the retrieved list (or none if absent):

With ranks (1, 2, 1, 5, absent) for five questions: recall@1 = 0.4, recall@3 = 0.6, recall@5 = 0.8, MRR = (1 + 0.5 + 1 + 0.2 + 0)/5 = 0.54. Recall@k is the quantity that bounds the system: if the gold passage is not among the k passages in the prompt, the model has to guess.

Composition of errors

Let r be the recall@k, a the accuracy of the generator when the gold passage is in the prompt and b its accuracy when it is not (including correct refusals). By the law of total probability:

With r = 0.9, a = 0.85, b = 0.2: 0.9 · 0.85 + 0.1 · 0.2 = 0.785. Improving retrieval to r = 0.95 gives 0.8175; improving the generator to a = 0.95 gives 0.875. The decomposition tells you where the next hour of work should go, and it needs the two measurements separately.

Prompt budget

The context costs about 1.3 tokens per English word. Four chunks of 200 words are about 1,040 tokens per question; at long prompts the prefill time and the price grow in proportion (Lesson LM10-T5). More passages are not free and can distract the model: accuracy often falls when many irrelevant chunks are added.

A worked example: a tiny end-to-end retriever

The code counts chunks, builds a TF-IDF index over four chunks, retrieves for two questions, and computes recall and MRR from ranks and the error composition. Notebook LM11-L3 extends this to a full prototype with prompt assembly and an injection test.

import math

def nchunks(nwords, size, overlap):
    if nwords <= size:
        return 1
    return math.ceil((nwords - size) / (size - overlap)) + 1

print(nchunks(1000, 200, 50), nchunks(1000, 200, 0), nchunks(150, 200, 50))   # 7 5 1

def tokens(text):
    return ''.join(ch if ch.isalnum() else ' ' for ch in text.lower()).split()

def build(chunks):
    n, df = len(chunks), {}
    toks = [tokens(c) for c in chunks]
    for t in toks:
        for w in set(t):
            df[w] = df.get(w, 0) + 1
    idf = {w: math.log((1 + n) / (1 + d)) + 1 for w, d in df.items()}
    vecs = []
    for t in toks:
        v = {}
        for w in t:
            v[w] = v.get(w, 0) + idf[w]
        vecs.append(v)
    return idf, vecs

def search(idf, vecs, question, k):
    q = {}
    for w in tokens(question):
        if w in idf:
            q[w] = q.get(w, 0) + idf[w]
    qn = math.sqrt(sum(x * x for x in q.values()))
    out = []
    for i, v in enumerate(vecs):
        vn = math.sqrt(sum(x * x for x in v.values()))
        s = sum(x * v.get(w, 0.0) for w, x in q.items()) / (qn * vn) if qn and vn else 0.0
        out.append((s, i))
    out.sort(key=lambda p: (-p[0], p[1]))
    return [(i, round(s, 3)) for s, i in out[:k]]

chunks = ["The dock is in bay four.", "Robots dock when the battery is low.", "A full charge takes ninety minutes.", "Visitors wear vests."]
idf, vecs = build(chunks)
print(search(idf, vecs, "Where is the dock?", 2))              # [(0, 0.619), (1, 0.564)]
print(search(idf, vecs, "How long does a full charge take?", 2))   # [(2, 0.707), (0, 0.0)]

def recall_at_k(ranks, k):
    return sum(1 for r in ranks if r is not None and r <= k) / len(ranks)

def mrr(ranks):
    return sum(1 / r for r in ranks if r is not None) / len(ranks)

ranks = [1, 2, 1, 5, None]                  # rank of the gold passage per question; None = not retrieved
print(recall_at_k(ranks, 1), recall_at_k(ranks, 3), recall_at_k(ranks, 5), round(mrr(ranks), 2))   # 0.4 0.6 0.8 0.54

r, a, b = 0.9, 0.85, 0.2                    # recall, accuracy with the gold passage, accuracy without it
print(round(r * a + (1 - r) * b, 3))      # 0.785

The second question has a score of exactly 0 for the runner-up because that chunk shares no word with the question. The ranking is still well defined; a threshold on the best score is how a system decides it has nothing to offer.

Debugging a RAG system

Engineering implications and limitations

Evaluate retrieval on its own. Build a set of real questions with known gold passages (30 to 200) and report recall@k and MRR with intervals (Module 9). Do this before choosing the generator.

Include unanswerable questions. A system that always answers will hallucinate when retrieval has nothing. Measure the rate of correct refusals separately and choose a similarity threshold or an explicit “not found” behaviour from that data.

Ask for citations and check them. Require passage numbers in the answer; verify programmatically that the numbers exist and, where feasible, that quoted strings occur in the cited passage. Show sources to the user.

Filter by permission before retrieval. Access control must be part of the search, not a clean-up after generation, or the model will have seen content the user may not.

Keep the index fresh. Store document versions and timestamps, re-index on change, and delete removed documents. A stale index is a silent source of wrong answers.

Treat retrieved text as untrusted. Documents can contain instructions aimed at the model. Delimit them, instruct the model to treat them as data, and never let their content trigger actions (Lesson LM11-T6).

For robots. A robot assistant that answers from manuals should cite the section and show it to the operator, and it must refuse rather than guess for safety-relevant questions (limits, procedures). Answers from retrieval can inform a human; they should not directly set parameters without validation.

Common misconceptions

Engineering takeaways

1. RAG supplies relevant passages at question time: chunk and index offline; retrieve, assemble a delimited prompt and generate online.
2. Chunks needed: ⌈(n − s)/(s − o)⌉ + 1; overlap keeps boundary facts whole at the price of a larger index.
3. Recall@k and MRR evaluate retrieval; end-to-end accuracy = r·a + (1 − r)·b separates retrieval from generation.
4. Design for unanswerable questions: threshold or explicit refusal, citations that are verified, and quote checks.
5. Index freshness, permission filtering and hostile documents are operational risks that need owners and tests.

References and further reading

The numbers were computed by the code in this lesson. The cost estimate of 1.3 tokens per word is a rule of thumb for English text and depends on the tokenizer (Module 2).

• Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv:2005.11401.
• Karpukhin, V. et al. (2020). Dense passage retrieval for open-domain question answering. arXiv:2004.04906.
• Greshake, K. et al. (2023). Not what you have signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. arXiv:2302.12173.
• Lewis et al. (2020), RAG (research paper)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: tool calling and schema validation: letting a model request actions while the application stays in control.

Related content

Nothing below is required to finish this lesson.

• Before
  LM11-T2: Embeddings and Semantic Similarity

  Represent text as vectors and rank by cosine similarity.

• Lab
  LM11-L1: Chunking and Retrieval

  Chunk documents, build TF-IDF vectors and retrieve the best passages for a question.