Skip to the lab
FixTheRobot Explore Robotics / LLM / LM11 / LM11-L3

LLM · Notebook lab · LM11 · Intermediate

LM11-L3 — Build a Retrieval Question-Answering Prototype

Build the retrieval half of a RAG system, measure it, and see exactly what a hostile document does to it.

  • Module 11: Building LLM Applications
  • PyTorch notebook
  • Hardware: CPU only, about 2 GB of RAM, a few minutes (retrieval and prompt assembly; the generation step is pluggable)
  • About 60 to 90 minutes

The browser labs of this module chunked text and validated tool calls on fixed examples. Here you assemble a small end-to-end pipeline from documents to a prompt, and you test it the way you would test a real one: retrieval first, then answers, then an attack.

The reader is a deterministic extractive function, not a language model, so the notebook runs anywhere and its results are exactly reproducible. A real model plugs in at one clearly marked place.

Objective

Implement chunking, TF-IDF retrieval, recall@k, a delimited prompt builder and a passage screen so that all five self-checks pass, and explain from your measurements which settings help and why screening is not a sufficient defence against prompt injection.

Why a notebook? This exercise trains or runs a real PyTorch model. PyTorch does not run in the browser Python used by the site’s other labs, so you run this one on your own computer or in a hosted notebook service. The interactive browser labs of this module cover the surrounding ideas without PyTorch.

Setup and requirements

Hardware: CPU only, about 2 GB of RAM, a few minutes (retrieval and prompt assembly; the generation step is pluggable). CPU only; about 2 GB of RAM; standard library only (no PyTorch needed). Everything runs in well under a second.

Download the notebook (.ipynb) Reference solution (try the lab first)

On your own machine (Python 3.10 or newer):

python3 -m venv llm-env
source llm-env/bin/activate        # on Windows: llm-env\Scripts\activate
pip install torch numpy jupyterlab
jupyter lab retrieval-qa-prototype.ipynb

In a hosted notebook service such as Google Colab or Kaggle, upload the file and run it there; PyTorch is usually preinstalled. Matplotlib is used for optional plots when it is available; the notebook runs without it.

What you will do

  1. Chunking. Split a text into overlapping word chunks and test the edge cases.
  2. Index and retrieval. Compute idf and TF-IDF vectors and rank chunks by cosine similarity.
  3. Measure retrieval. Compute recall@k for whole documents and for 12-word chunks, before any generation.
  4. Prompt assembly. Build a prompt with an instruction, delimited numbered passages and the question.
  5. Answers with and without context. Compare a pluggable reader with and without retrieved passages.
  6. Injection test. Retrieve a hostile document, flag it with a screening step and keep its claims out of the prompt.

Cells that you must complete are marked TODO in the notebook and have a self-check right after them (an assert with a message). Run the cells from the top; a failing check tells you what is wrong.

Expected behaviour

  • Self-check 1: chunking gives a b c / c d e / e f g for 7 words with size 3 and overlap 1, handles short and empty texts.
  • Self-check 2: idf values match the formula on a three-chunk example and each obvious question finds its document.
  • Self-check 3: with whole documents recall@1 and recall@3 are both 1.000; with 12-word chunks (30 chunks) recall@1 is 0.625 and recall@3 is 0.875 for the reference solution.
  • Self-check 4: the prompt has exactly the specified lines and delimiters.
  • Section 6: with context the extractive reader contains the gold answer for 75% of the questions, without context for 0%. Self-check 5: the hostile passage is retrieved, flagged and absent from the prompt.

How to know you are done

  • All five self-checks print “passed” when the notebook is run from the top.
  • You can explain why recall@k is measured before answer quality and what a low recall@k tells you to change.
  • You can name two ways in which a rephrased attack would defeat the phrase screen, and what the application must do instead.

The site cannot run or grade the notebook for you. The self-checks inside it are your verification. The reader is an extractive stand-in and not a language model; no generation quality is measured. The numbers given above were produced by the reference solution when this module was prepared.

If something goes wrong

  • Chunks do not match: the last chunk must stop at the end of the text, and the step between starts is size − overlap, not size.
  • Scores differ in the last decimals: compare with a tolerance; the order of summation changes the last bits.
  • Recall is lower than expected: check that tokenisation lower-cases and removes punctuation for both the chunks and the question.
  • Python version: any Python 3.8 or newer works; nothing outside the standard library is used.

Theory for this lab

Reference material

    Building LLM Applications labs

    1. LM11-L1: Chunking and Retrieval
    2. LM11-L2: Validate Structured Outputs and Tool Calls
    3. LM11-L3: Build a Retrieval Question-Answering Prototype: this lab.

    Next lab: LM12-L1: Capstone Milestone 1: Dataset, Tokenizer and Pipeline.