M3.2 — Embeddings and Tokenization

Learning objectives

After this lesson, you should be able to:

• Explain tokenization and the role of a vocabulary.
• Describe why subword tokens, such as those from byte-pair encoding, are used.
• Explain an embedding as a learned lookup table and as a matrix multiplication.
• Compute cosine similarity between two vectors.
• Describe how image patches and robot actions can also become tokens.
• Read the shapes in an nn.Embedding example.

The one-sentence idea

Tokenization turns an input into integer ids from a fixed vocabulary, and an embedding table turns each id into a learned vector the model can compute with.

Ids are labels, and embeddings are where meaning is learned.

The problem: models only take numbers

A neural network multiplies vectors by matrices (M1.1). It cannot take a sentence, a camera image or a robot command as they are. Something has to turn each kind of input into a sequence of vectors. That happens in two stages: tokenization cuts the input into discrete pieces called tokens and gives each an integer id, and an embedding turns each id into a vector the model can compute with.

Together they are the front door of every language model, and, as the later sections show, the same idea is reused for image patches and for robot actions in VLA models.

Tokenization: from text to integer ids

The simplest tokenizer splits on spaces and numbers each distinct word. The mapping from token to id is the vocabulary.

text = "pick up the red cup"
words = text.split()
vocab = {w: i for i, w in enumerate(sorted(set(words)))}
ids = [vocab[w] for w in words]
print(vocab)
print(ids)
print(vocab.get("blue", -1))     # a word the vocabulary has never seen

{'cup': 0, 'pick': 1, 'red': 2, 'the': 3, 'up': 4}
[1, 4, 3, 2, 0]
-1

The last line shows the weakness of word-level tokens: a word outside the vocabulary has no id, and a vocabulary that covers every word, name and typo in a language would be enormous. Splitting into single characters solves that but makes sequences very long.

Modern models use subword tokens as a compromise. Frequent words stay whole, and rare words are built from pieces. A popular method, byte-pair encoding (BPE), starts from single characters and repeatedly merges the most frequent adjacent pair into a new token. This small version applies it to the string aaabdaaabac:

from collections import Counter

def merge_most_frequent(tokens):
    pairs = Counter(zip(tokens, tokens[1:]))
    first = {}
    for i, p in enumerate(zip(tokens, tokens[1:])):
        first.setdefault(p, i)
    best = max(pairs, key=lambda p: (pairs[p], -first[p]))   # most frequent, earliest on ties
    out, i = [], 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == best:
            out.append(tokens[i] + tokens[i + 1]); i += 2
        else:
            out.append(tokens[i]); i += 1
    return best, pairs[best], out

tokens = list("aaabdaaabac")
for _ in range(3):
    best, count, tokens = merge_most_frequent(tokens)
    print(best, count, tokens)

('a', 'a') 4 ['aa', 'a', 'b', 'd', 'aa', 'a', 'b', 'a', 'c']
('aa', 'a') 2 ['aaa', 'b', 'd', 'aaa', 'b', 'a', 'c']
('aaa', 'b') 2 ['aaab', 'd', 'aaab', 'a', 'c']

Each merge adds one entry to the vocabulary, and the string becomes shorter in tokens. A real tokenizer learns tens of thousands of merges from a large body of text. Because any word can fall back to smaller pieces, there is never a word it cannot encode, and the result is that sentences become short sequences of integer ids from a fixed vocabulary.

Embeddings: from ids to vectors

An integer id has no useful arithmetic: token 4 is not “twice” token 2. An embedding table fixes that. It is a matrix with one row per vocabulary entry, and looking up a token’s row gives its vector. The size of each row is the embedding dimension.

import numpy as np

E = np.arange(15, dtype=float).reshape(5, 3)    # vocabulary of 5, dimension 3 (placeholder values)
ids = [1, 4, 3, 2, 0]
print(E[ids].shape)                              # one row per token

one_hot = np.eye(5)[ids[0]]                     # a vector of zeros with a single 1
print(one_hot)
print(np.allclose(one_hot @ E, E[ids[0]]))

(5, 3)
[0. 1. 0. 0. 0.]
True

The last line shows that a lookup is the same as multiplying a one-hot vector by the table, so an embedding is just a matrix multiplication (M1.1) that is implemented as a faster lookup. The values in the table are learned parameters: they start random and are trained together with the rest of the model, so that tokens used in similar ways end up with similar vectors.

“Similar” is usually measured with cosine similarity, the dot product of two vectors divided by their lengths. It is 1 for vectors pointing the same way and near 0 for unrelated directions. The three vectors below are hand-made to show the idea, not taken from a trained model:

cat = np.array([1.0, 0.9, 0.0])
dog = np.array([0.9, 1.0, 0.1])
car = np.array([0.0, 0.1, 1.0])

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(round(cosine(cat, dog), 3), round(cosine(cat, car), 3))

0.992 0.067

In a trained model the geometry is learned from data rather than written by hand, and it is far higher dimensional, but the principle holds: a good embedding places related things close together. You saw this idea in the dot-product similarity of M1.1, and it is the basis of CLIP, where an image and its caption are pushed towards nearby vectors.

import torch
from torch import nn

emb = nn.Embedding(5, 3)                 # 5 tokens, dimension 3
out = emb(torch.tensor([1, 4, 3]))
print(out.shape, emb.weight.shape)
print(torch.equal(out, emb.weight[[1, 4, 3]]))

torch.Size([3, 3]) torch.Size([5, 3])
True

Tokens beyond text: patches and actions

The token idea is not limited to words. Models that handle images and robots use it too:

• Image patches. A Vision Transformer cuts an image into a grid of small squares and treats each as a token, after flattening it to a vector. A 224 × 224 image with 16 × 16 patches gives 14 × 14 = 196 patches, each with 16 × 16 × 3 = 768 numbers.
• Action tokens. Some VLA models turn each continuous action value into one of a fixed number of bins, so that the action can be predicted like the next word. Phase 4 of the roadmap covers the tradeoffs.

print((224 // 16) ** 2, 16 * 16 * 3)

a = np.array([-1.0, 0.0, 0.5])                         # actions scaled to [-1, 1]
tokens = np.clip(np.round((a + 1) / 2 * 255), 0, 255).astype(int)
print(tokens)
print((tokens / 255 * 2 - 1).round(3))                 # decoding back

196 768
[  0 128 191]
[-1.     0.004  0.498]

The decoded values are close to the originals but not identical, because binning discards detail, so the number of bins sets how precisely the robot can be commanded. The point for now is only that one tokenize-then-embed pattern can serve text, images and actions.

Common misconceptions

Engineering takeaways

1. A tokenizer cuts input into tokens and maps each to an integer id from a vocabulary.
2. Word-level vocabularies cannot handle unseen words. Subword methods such as BPE can.
3. BPE repeatedly merges the most frequent adjacent pair into a new token.
4. An embedding table has one learned row per token, and a lookup equals a one-hot vector times the table.
5. Cosine similarity measures how closely two vectors point the same way.
6. Embeddings are learned so that related tokens end up close together.
7. Image patches and discretised actions can be tokens too.
8. The number of action bins limits how precisely a tokenised action can be decoded.

Everything a model reads has to become a sequence of vectors first, and tokens and embeddings are how.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You can now turn input into a sequence of vectors. Next is the mechanism that lets each vector in that sequence look at all the others.

Back to the VLA roadmap prerequisites