LM2-T2 — Characters, Words, Subwords and Token IDs

Learning objectives

After this lesson, you should be able to:

• Compare character, word and subword tokenization by vocabulary size, sequence length and coverage.
• Explain what a vocabulary and a token ID are, and how an ID selects a row of the embedding table.
• Estimate the effect of vocabulary size on the embedding parameters and of sequence length on cost.
• Write a minimal encoder and decoder around a vocabulary.

The idea: pieces that are neither too small nor too big

A token is the unit a model reads. A vocabulary is the fixed list of tokens it knows; each gets an integer ID, usually its position in the list. The text “the robot picks up the red box” becomes, for example, [262, 9379, 6730, 510, 262, 2266, 3091]: not meaningful by itself, only an index.

Characters

A vocabulary of letters, digits and punctuation has only a hundred or so entries, and nothing is ever unknown. But sequences are long, about 4 to 5 times the number of words, and attention cost grows with the square of the length (Lesson LM3-T6). A character model also has to learn spelling from scratch before it can learn meaning.

Words

One token per word gives short sequences and meaningful units. But the vocabulary must contain every form: “pick”, “picks”, “picked”, “picking”, and every name and typo. English has hundreds of thousands of word forms; most are rare (Zipf’s law, Lesson LM2-T1). Any word not in the list becomes <unk>, losing information, and rare words get embeddings trained on a handful of examples.

Subwords

Subword tokenization keeps frequent words whole and breaks rare words into frequent pieces: “picks” may be “pick” + “s”, a name like “Raschka” a few pieces. The vocabulary is a fixed size (tens of thousands), sequences are only slightly longer than word sequences, and every string can be encoded because single characters (or bytes) are always available as a last resort. Byte-pair encoding (Lesson LM2-T3) is the standard method.

IDs are indexes

The ID has no meaning as a number: 262 is not “bigger than” 261 in any useful sense. The model uses it only to look up a row of a learned table, the embedding matrix, which holds one vector per token (Lesson LM3-T1). That is why the mapping from token to ID must never change after training: the row for ID 262 was trained for a specific token.

The mathematics: vocabulary size, sequence length and embeddings

Let V be the vocabulary size, d the width of an embedding vector, and n the number of tokens in an input. The embedding table is a matrix E of shape V × d, and the input token IDs t₁, …, t_n select rows:

V

vocabulary size: the number of distinct tokens, including special tokens.

d

embedding width, e.g. 768 for GPT-2 small.

t_i

the integer ID of the i-th token, with 0 ≤ t_i < V.

x_i

the d-dimensional vector for that token; row t_i of E.

The embedding table has V · d parameters, and (in GPT-2) the output layer that produces the next-token logits is a matrix of the same shape, so a large vocabulary is paid for twice. For V = 50,257 and d = 768, V · d = 38,597,376.

Trade-off. If a text of m words needs L tokens, the compression is L/m tokens per word. A larger vocabulary lowers L/m and raises V · d. Attention costs about n² per layer, so halving the sequence length cuts that cost by about 4 times; the output layer costs about n · d · V, which grows with V. Real vocabularies of 30,000 to 100,000 tokens balance the two.

A worked example: one sentence three ways

The sentence “the robot picks up the red box” (30 characters including spaces, 7 words):

For a 1,000-word document: about 5,000 characters, 1,000 words or about 1,300 subword tokens (using the rule of thumb of three quarters of a word per token). Attention work scales with the square: roughly 25,000,000 pair interactions for characters, 1,000,000 for words, 1,700,000 for subwords. The subword sequence costs only 1.7 times the word sequence and 15 times less than characters.

Embedding size. With d = 768 the character vocabulary of 100 needs 100 × 768 = 76,800 embedding parameters; a word vocabulary of 200,000 needs 153.6 million, more than all of GPT-2 small; the subword vocabulary of 50,257 needs 38.6 million.

A minimal encoder and decoder in Python

text = "the robot picks up the red box"
words = text.split()
vocab = {w: i for i, w in enumerate(sorted(set(words)))}   # token -> id
inverse = {i: w for w, i in vocab.items()}                 # id -> token

ids = [vocab[w] for w in words]
print(ids)                                    # [5, 3, 2, 4, 5, 1, 0]
print(" ".join(inverse[i] for i in ids))      # the robot picks up the red box

Sorting the set of words gives a reproducible numbering: box 0, red 1, picks 2, robot 3, up 4, the 5. The encoder looks each word up; the decoder uses the inverse dictionary. A word that is not in vocab raises a KeyError, the weakness that special tokens (Lesson LM2-T4) and subwords repair.

Engineering implications and limitations

The tokenizer ships with the model. A trained model is meaningless without the exact tokenizer and vocabulary it was trained with. Save them together and version them.

Count tokens with the real tokenizer. Costs, limits and speed are per token. The number of tokens per word depends on the language and on content such as code, numbers or rare names.

Watch non-English text and numbers. A tokenizer trained mostly on English may split other languages and long numbers into many small pieces, making them costlier and harder for the model to handle.

Do not interpret IDs. IDs are arbitrary labels. Never compute with them as if they were quantities.

Choose the vocabulary size deliberately. A small tokenizer is easier to train and a small model needs a small embedding table; this track uses tiny vocabularies for tiny models.

For robots. Command vocabularies for a robot (part numbers, coordinates, units) often tokenize badly with a general tokenizer: “A7-12” may become many pieces. Prefer structured outputs with fixed fields (Module 11) over free text for identifiers and numbers.

Common misconceptions

Engineering takeaways

1. A vocabulary maps tokens to integer IDs; IDs are indexes into the embedding table, not quantities.
2. Characters: tiny vocabulary, long sequences. Words: short sequences, huge vocabulary, unknowns. Subwords: the practical compromise with no unknowns.
3. The embedding table has V × d parameters (38.6 million for V = 50,257, d = 768), and the output layer is usually the same size.
4. Sequence length matters because attention cost grows with its square; subword tokens run about 1.3 per English word.
5. Keep the tokenizer and the vocabulary together with the model and never change the mapping after training.

References and further reading

The core reference is Chapter 2 of the book (sections on tokenizing text and converting tokens to IDs).

• Sennrich, R., Haddow, B. and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909.
• Build a Large Language Model (From Scratch), Chapter 2: Working with Text Data (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: byte-pair encoding, the algorithm that builds a subword vocabulary from data.

Related content

Nothing below is required to finish this lesson.

• Before
  LM2-T1: Text Normalization and Preprocessing

  Decide what cleaning a text corpus needs and what each choice removes or keeps.

• Lab
  LM2-L1: Build a Simple Tokenizer

  Write encode and decode for a word-level vocabulary with special tokens and inspect the token IDs.