LM2-T4 — Special Tokens and Unknown Text

Learning objectives

After this lesson, you should be able to:

• Explain the roles of the end-of-text, padding and unknown tokens.
• Describe why a word-level tokenizer needs an unknown token and a byte-level one does not.
• Join several documents into one training stream with separators and explain what goes wrong without them.
• Use special tokens safely in code, including the allowed_special rule of tiktoken.

The idea: tokens that are about the text, not in the text

Ordinary tokens stand for pieces of language. Special tokens stand for events in the pipeline. The vocabulary reserves IDs for them, and the embedding table has rows for them like any other token, but they never appear in a normal sentence.

End-of-text

Training data is made of many documents: web pages, books, emails. For efficiency they are joined into one long stream and cut into fixed-length windows (Lesson LM2-T5). Without a marker the model would see the last sentence of one article followed directly by the first sentence of an unrelated one, and learn that such jumps happen mid-thought. An end-of-text token, written <|endoftext|> in GPT-2, is inserted between documents. At generation time it has a second use: when the model predicts it, the answer is finished, and generation can stop (Lesson LM5-T2). A model that was never taught to produce it has no natural way to stop.

Padding

A batch is a rectangular array, but real examples differ in length. Shorter ones are filled up with a padding token. The padding is not data: the model should not predict it and should not attend to it. We therefore keep a mask that marks real positions, and exclude padded positions from the loss. For pretraining on long streams, windows all have the same length and no padding is needed; for fine-tuning on short examples (Modules 6 and 7) padding is routine.

Unknown

A word-level vocabulary cannot contain every word, so words outside it are replaced by an unknown token, <unk>. The model then sees the same symbol for “Zürich”, “quokka” and a typo, and the information is gone. Subword tokenizers rarely need it, and byte-level ones never do.

Other special tokens

Chat and instruction models add more: markers for the beginning of a sequence, for the start and end of each role or turn, and sometimes for tool calls. They work the same way: reserved IDs with learned embeddings that the application inserts around ordinary text. Different model families use different markers, so a chat template belongs to the model (Lesson LM7-T1).

The mathematics: masks, loss on real tokens, and vocabulary accounting

Let a batch have B sequences padded to length T. Let m_b,t be 1 if position t of sequence b holds a real token and 0 if it is padding. The average loss over real positions is

B, T

batch size (number of sequences) and padded length (tokens per sequence).

m_b,t

the mask: 1 for a real token, 0 for padding; a tensor of shape (B, T).

ℓ_b,t

the cross-entropy loss of the prediction at position t of sequence b; shape (B, T) before masking.

L

a single number: the mean loss per real token.

Dividing by the number of real tokens, not by B · T, matters: if half of the positions are padding and you divide by B · T, the loss is halved and the learning signal is distorted. In PyTorch, cross_entropy can skip positions whose target equals a chosen ignore_index, commonly −100, which is how the book handles padding in fine-tuning.

Vocabulary accounting. Special tokens occupy IDs just like ordinary ones: V = V_text + S where S is the number of special tokens. GPT-2: 50,256 text tokens (256 bytes plus 50,000 merges) plus one end-of-text token gives 50,257, and the end-of-text token has the last ID, 50,256. Adding a token later increases V, so the embedding table and output layer must grow by S · d parameters each.

Unknown rate. For a word-level tokenizer the fraction of tokens that become unknown on new text is u = (number of unknown tokens) / (number of tokens). A vocabulary of the 1,000 most frequent words may cover well over 80% of running text, yet the uncovered few percent contain most of the content words. For byte-level BPE, u = 0 by construction, since 256 byte symbols cover all inputs.

A worked example: documents, padding and unknowns

Joining documents. Documents with 4, 3 and 5 tokens joined with separators give 4 + 1 + 3 + 1 + 5 + 1 = 15 tokens. A window of length 5 cut from position 3 contains the tail of document A, its separator, and the first three tokens of document B. The model learns that after the separator comes a fresh start.

Padding and the loss mask. Two sequences with real lengths 3 and 5 are padded to length 5. The mask has shape (2, 5): [1,1,1,0,0] and [1,1,1,1,1]. If every real position has loss 2.0, the correct mean is 16 / 8 = 2.0 over 8 real tokens, but dividing by B · T = 10 would give 1.6.

import math

lengths = [3, 5]
T = max(lengths)
mask = [[1] * n + [0] * (T - n) for n in lengths]
loss = [[2.0] * T for _ in lengths]            # every position has loss 2.0 (padding included)

total = sum(m * l for mr, lr in zip(mask, loss) for m, l in zip(mr, lr))
real = sum(sum(r) for r in mask)
print(mask)                                    # [[1, 1, 1, 0, 0], [1, 1, 1, 1, 1]]
print(round(total / real, 3), round(total / (len(lengths) * T), 3))   # 2.0 1.6

Unknown words. Take a vocabulary built from the words “the”, “robot”, “picks”, “up”, “box” and the text “the quokka picks up the box”. A word-level encoder replaces “quokka” by <unk>: the model cannot tell what was picked up beyond the box. One unknown of six tokens is an unknown rate of 1/6, about 17%. A byte-level tokenizer encodes “quokka” as a few pieces such as “qu”, “ok”, “ka” (exact pieces depend on the vocabulary) and loses nothing.

Special tokens with tiktoken

The book encodes text with tiktoken. A string that looks like a special token is, by default, not converted; the library raises an error unless you opt in. That default protects you from users who type <|endoftext|> into a prompt to confuse the model. In code, the usual pattern is:

import tiktoken   # not run in this site: requires the package and its vocabulary file (downloaded once)

tok = tiktoken.get_encoding("gpt2")
ids = tok.encode("Hello <|endoftext|> world", allowed_special={"<|endoftext|>"})
print(ids)                                   # the special token appears as one ID, 50256
print(tok.decode(ids))                       # Hello <|endoftext|> world

Note the comment: the first call downloads the vocabulary files, so it needs network access the first time. The track’s notebooks that need tiktoken say so in their setup notes and offer an offline path.

Engineering implications and limitations

Always separate documents. When you join documents for pretraining, insert an end-of-text token between them. Leaving it out is a quiet bug: nothing fails, training loss looks fine, and the model merges unrelated texts.

Mask the padding. Padded positions must be excluded from the loss (and from attention in architectures that need it). If you use a real token ID for padding and do not mask it, the model is rewarded for predicting padding and learns to output it.

Stop on the end token. Generation loops need a stop condition. Check for the end-of-text ID and the maximum length; without both, a model that never emits the end token runs to the limit, and one that always does returns empty answers.

Do not let user text create special tokens. If your application passes user input to a tokenizer with special tokens allowed, a user can inject control tokens. Encode user text with special tokens disallowed, and add the markers yourself around it.

Keep IDs stable. Special tokens live at fixed IDs. Changing the order of the vocabulary or adding tokens in the middle breaks every trained checkpoint.

Measure the unknown rate. For any word-level tokenizer, compute the unknown rate on held-out text and on the kind of text your application receives (names, part numbers, new terms). A low rate on training data says nothing about what the robot will meet.

For robots. A command interface should never rely on free text for identifiers. A unknown token in a part number is a wrong action waiting to happen; validate model outputs against a schema (Module 11).

Common misconceptions

Engineering takeaways

1. End-of-text separates documents in the training stream and lets generation stop; padding fills batches and must be masked; unknown replaces what a vocabulary lacks.
2. Special tokens are reserved IDs with learned embeddings, counted in the vocabulary size (GPT-2: 50,256 text tokens plus one special = 50,257).
3. The loss must average over real tokens only: Σ m·ℓ divided by Σ m.
4. Byte-level BPE has no unknown token because all text is bytes; word-level tokenizers have an unknown rate that must be measured.
5. Disable special-token parsing on user text, and add markers in code.

References and further reading

See Chapter 2 of the book for the end-of-text and unknown tokens in the simple tokenizer and for the tiktoken GPT-2 encoding.

• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI technical report; the tokenizer behaviour described here is shown in Chapter 2 of the book.
• Build a Large Language Model (From Scratch), Chapter 2: Working with Text Data (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: turning a token stream into input and target pairs with a sliding window.

Related content

Nothing below is required to finish this lesson.

• Before
  LM2-T3: Byte-Pair Encoding

  Learn merge rules from a corpus and apply them to encode new text.

• Lab
  LM2-L2: Implement Byte-Pair Encoding

  Learn merges from a tiny corpus and encode text with them.