LM11-T2 — Embeddings and Semantic Similarity

Learning objectives

After this lesson, you should be able to:

• Explain how a text becomes a vector in sparse (TF-IDF) and dense (learned) form.
• Compute dot product, norm and cosine similarity by hand and in code.
• Describe nearest-neighbour search and estimate the memory of an index.
• List what embeddings handle poorly and how to evaluate a choice with recall.

The idea: meaning as position

In Module 3 each token got a vector in a learned space, and tokens used in similar ways ended up close together. The same idea works for whole passages: map a sentence, a paragraph or a question to a single vector so that texts that mean similar things get similar vectors. Then “find passages relevant to this question” becomes “find the stored vectors nearest to the question’s vector”, a geometric problem that computers solve quickly.

Two families of vectors

Sparse vectors (bag of words, TF-IDF) have one dimension per vocabulary word, almost all zero. They match words: a passage about “battery charging” matches the question “where do robots charge?” only through shared words. They are cheap, need no training, are easy to explain, and work very well when questions reuse the documents’ vocabulary (names, part numbers, error codes). They fail on paraphrase: “power up the robot” and “recharge the machine” share no words.

Dense vectors have a few hundred to a few thousand real-valued dimensions, all non-zero. They are produced by a neural embedding model, typically a transformer encoder (or a decoder used in a particular way) that reads the text and pools its hidden states, for example by averaging them, into one vector. The model is trained so that matching pairs, such as a question and a passage that answers it, or two paraphrases, get high similarity and non-matching pairs get low similarity (contrastive training). They match meaning to a useful extent, including paraphrase and some cross-language similarity.

Which to use

Neither is always better. Dense retrieval wins on paraphrase and loosely worded questions; sparse retrieval wins on exact identifiers and rare terms; many systems combine them (hybrid retrieval) by merging the two ranked lists. The way to choose is to measure recall@k on your own questions (Module 9 methods, Lab LM11-L1).

What an embedding cannot do

• Negation. “The robot is allowed in the corridor” and “The robot is not allowed in the corridor” usually have very similar vectors, because they are about the same thing.
• Numbers and exact values. “limit of 2 kg” and “limit of 20 kg” are nearly identical to most embedding models.
• Domain terms. A general model may place your internal part names badly. Test on your data.
• Answerability. The nearest passage is returned even when no passage answers the question. Similarity is not truth.

Similarity is a ranking, not a probability

The mathematics: dot product, norm, cosine

For vectors a and b in R^d:

a·b

dot product: large when the vectors point the same way and are long.

‖a‖

length of the vector.

cos(a, b)

the cosine of the angle between a and b; 1 for the same direction, 0 for perpendicular, −1 for opposite.

Worked example. a = (1, 2, 0, 1), b = (2, 4, 0, 2), c = (0, 1, 3, 0). Then ‖a‖ = √6 = 2.4495. b = 2a, so cos(a, b) = 1.0: same direction, different length, maximal similarity. a·c = 2, ‖c‖ = √10, so cos(a, c) = 2/(√6·√10) = 0.2582. And cos((1, 0), (−1, 0)) = −1.

Normalise once, then use the dot product

If every stored vector and the query are scaled to length 1 (â = a/‖a‖), then cos(a, b) = â·b̂. Normalising at indexing time turns similarity search into plain matrix multiplication, which hardware does extremely fast. Euclidean distance on normalised vectors gives the same ranking: ‖â − b̂‖² = 2 − 2·cos(a, b).

Search and memory

Brute-force search computes one dot product per stored vector: N · d multiply-adds per query. The index itself takes N · d · b bytes. For one million passages with d = 768 in 32-bit floats: 1,000,000 · 768 · 4 = 3.07 GB; in 8 bits 0.77 GB. Beyond some millions of vectors, systems use approximate nearest-neighbour indexes (graph-based or clustering-based) that examine only a small part of the collection and return the true nearest neighbours most of the time, trading a little recall for large speed-ups. Their recall must be measured like any other component.

TF-IDF as a sparse embedding

For a term t in a text, the weight is count(t) · idf(t) with idf(t) = ln((1 + N)/(1 + df(t))) + 1, where df is the number of chunks containing t (Lab LM11-L1). Rare terms get large weights, common ones small, and the cosine of two such vectors is a sound baseline against which every dense model must prove itself.

A worked example: similarity and search

The code reproduces the numbers above, shows that normalising first gives the same cosine, runs brute-force top-k search over a few hand-made vectors, and computes the index memory. The vectors are made up for illustration: a real embedding model would produce them from text.

import math

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def norm(a):
    return math.sqrt(dot(a, a))

def cos(a, b):
    return dot(a, b) / (norm(a) * norm(b))

a, b, c = [1.0, 2.0, 0.0, 1.0], [2.0, 4.0, 0.0, 2.0], [0.0, 1.0, 3.0, 0.0]
print(round(cos(a, b), 4), round(cos(a, c), 4), round(cos([1, 0], [-1, 0]), 4))   # 1.0 0.2582 -1.0
print(round(norm(a), 4))                                                          # 2.4495

na = [x / norm(a) for x in a]
nc = [x / norm(c) for x in c]
print(round(dot(na, nc), 4), round(cos(a, c), 4))                                 # 0.2582 0.2582

def top_k(query, vectors, k):
    scored = sorted(((round(cos(query, v), 9), i) for i, v in enumerate(vectors)), key=lambda p: (-p[0], p[1]))
    return [(i, round(s, 4)) for s, i in scored[:k]]

index = [a, b, c, [3.0, 0.0, 0.0, 1.0]]
print(top_k([1.0, 1.0, 0.0, 1.0], index, 2))        # [(0, 0.9428), (1, 0.9428)]: a and b tie, their lengths do not matter

print(1_000_000 * 768 * 4 / 1e9, 1_000_000 * 768 * 1 / 1e9)    # 3.072 0.768  (GB in float32 and in 8 bits)

The tie between a and b shows what cosine ignores: length. A passage and a ten-times-longer passage about the same thing can get the same score, which is a reason for chunking to a sensible size (Lesson LM11-T3).

Choosing and checking an embedding model

1. Write 30 to 100 real questions,With the passage (or the exact string) that answers each. Without these you can only guess.
2. Compute the TF-IDF baseline,Recall@1 and recall@5 with the simplest method (Lab LM11-L1).
3. Try one or two embedding models,Same chunks, same questions. Compare recall@k with intervals; check the maximum input length and the vector size (memory).
4. Look at the failures,Negation, numbers, identifiers, acronyms. They tell you whether hybrid retrieval or better chunks would help more than a bigger model.
5. Decide with cost in mind,Embedding dimension, model size, latency, licence and whether the data may leave your premises.

Engineering implications and limitations

Embed queries and passages the same way. Some models expect a prefix such as “query:” or “passage:” or a different template for each. Use the documented one; a mismatch quietly lowers recall.

Respect the model’s input limit. Text beyond the maximum length is cut off, so a long chunk is represented by its beginning only. Choose chunk sizes below the limit.

Store the model name and version with the index. Vectors from different models, or versions of one model, are not comparable. Changing the model means re-embedding everything.

Normalise once. Do it at indexing and at query time, or use an index that does it for you, and document which metric the index uses.

Keep sparse matching as a safety net. Exact identifiers, error codes and part numbers are often retrieved better by keyword search; a hybrid costs little.

Do not use a score threshold you did not measure. Pick “no answer” cut-offs from a labelled set including unanswerable questions.

Private data. Embeddings are derived from the text and can leak it in part. Treat the vector store with the same access controls as the documents.

Common misconceptions

Engineering takeaways

1. Texts become vectors; similar meaning should give a small angle. Cosine similarity cos(a, b) = a·b/(‖a‖‖b‖) ignores length; normalising first makes it a dot product.
2. Sparse TF-IDF vectors match words and need no training; dense vectors from embedding models match meaning, including paraphrase, and fail on negation and exact numbers.
3. Brute-force search costs N·d per query; an index of 10⁶ vectors of 768 float32 numbers takes 3.07 GB. Approximate indexes trade a little recall for speed.
4. A score is a ranking, not a probability. Measure recall@k on your own questions and compare against the TF-IDF baseline.
5. Embed queries and passages as documented, keep the model version with the index, and consider hybrid retrieval.

References and further reading

Lesson LM3-T1 introduces token embeddings. The vectors in the worked example are hand-made for illustration; the memory figures were computed by the code in this lesson.

• Mikolov, T. et al. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.
• Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv:1908.10084.
• Karpukhin, V. et al. (2020). Dense passage retrieval for open-domain question answering. arXiv:2004.04906.
• Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv:2005.11401.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: retrieval-augmented generation: chunk documents, retrieve passages and ground the answer in them.

Related content

Nothing below is required to finish this lesson.

• Before
  LM11-T1: Base Versus Instruction-Tuned Models and Prompt Design

  Choose a model type and write prompts and structured-output requests that are testable.

• Lab
  LM11-L1: Chunking and Retrieval

  Chunk documents, build TF-IDF vectors and retrieve the best passages for a question.