LM2-T1 — Text Normalization and Preprocessing

Learning objectives

After this lesson, you should be able to:

• List the main preprocessing steps for a text corpus and the information each one removes.
• Explain Unicode normalisation with a concrete example and apply it in Python.
• Compute the type-token ratio of a text and describe Zipf’s law.
• Decide which preprocessing a GPT-style pipeline needs and which it should avoid.

The idea: decide what counts as “the same”

Every preprocessing step answers the question: are these two strings the same thing? Is “The” the same as “the”? Is “café” typed with one character for é the same as with an e and a combining accent? Is a web page the same as itself with an extra cookie banner? If the answer is yes, the model sees one thing and learns from all the occurrences together; if no, it sees two things and each is learned separately, with fewer examples.

Common steps and what they cost

Modern practice: keep it raw, tokenize at the byte level

Early pipelines lowercased, stripped punctuation and removed rare words to keep vocabularies small. Current GPT-style systems mostly keep case, punctuation and whitespace as they are and use a byte-level tokenizer (Lessons LM2-T3 and LM2-T4), so that nothing is ever “unknown” and no information is thrown away before the model can judge its usefulness. The cleaning effort goes instead into data selection: removing duplicates, boilerplate and low-quality text, and checking the licence of the sources.

Why the tail matters

Zipf’s law says that in natural text the r-th most frequent word occurs roughly in proportion to 1/r. A few words (“the”, “of”, “and”) make up a large share of all tokens, and most words occur once or twice. A word-level vocabulary must either include the whole tail (huge, with poorly trained entries) or cut it (many unknowns). That is the reason subword tokenization exists.

The mathematics: type-token ratio and Zipf’s law

Count the number of tokens N (words with repeats) and the number of types V (distinct words). The type-token ratio is

N

the number of tokens in the text, counting every occurrence.

V

the number of distinct tokens (the vocabulary size of that text).

TTR

between 0 and 1: near 1 means almost every word is new; small means words repeat a lot. It falls as the text gets longer, because common words keep repeating while new words appear more slowly.

Zipf’s law says that the frequency f(r) of the word of rank r (rank 1 is the most frequent) is approximately

r

the rank of a word when words are sorted by frequency.

f(r)

the number of times that word occurs.

C

a constant (about the frequency of the most frequent word).

s

the exponent, close to 1 for natural-language text.

With s = 1 and C = 1000, the ten most frequent words have frequencies 1000, 500, 333, 250, 200, 167, 143, 125, 111, 100. The law is an empirical regularity, not an exact rule, and varies between languages and corpora.

Unicode normalisation. The same visible character can be encoded in more than one way. “é” can be one code point (U+00E9, precomposed) or two (U+0065 “e” followed by U+0301, a combining acute accent). The normal forms NFC (compose) and NFD (decompose) choose one representation; NFKC and NFKD also replace compatibility characters such as the ligature “fi” with plain letters. Without normalisation, two strings that look identical compare unequal and get different tokens.

A worked example: two ways to write “café”, and a type-token ratio

import unicodedata

a = "caf\u00e9"                              # "é" as one code point
b = unicodedata.normalize("NFD", a)         # "e" + combining accent
print(len(a), len(b), a == b)               # 4 5 False  (they look identical)
print(unicodedata.normalize("NFC", b) == a) # True  (normalising both to NFC makes them equal)

The two strings print the same on screen, but one has 4 code points and the other 5, so they are not equal. A byte-level tokenizer would encode them as different byte sequences. Normalising the corpus to one form (NFC is the usual choice) avoids learning two versions of every accented word.

Counting types and tokens. Take the text “the robot picks up the red box the robot picks up the green bin the arm picks up the blue tool”.

from collections import Counter

text = "the robot picks up the red box the robot picks up the green bin the arm picks up the blue tool"
words = text.split()
counts = Counter(words)
print(len(words), len(counts), round(len(counts) / len(words), 3))   # 21 tokens, 11 types, TTR 0.524
print(counts.most_common(3))                                         # [('the', 6), ('picks', 3), ('up', 3)]

There are N = 21 tokens and V = 11 types, so TTR = 11/21 = 0.524. “the” alone is 6 of the 21 tokens (29%), the signature of a Zipf-like distribution; 7 of the 11 types occur only once.

Preprocessing and what it implies for a robot. Suppose a robot’s voice interface logs commands with different spellings: “Pick up the Box”, “pick up the box ”, and “pick up the box.” (with trailing space or full stop). Without normalisation these are three different strings; after lowercasing, trimming whitespace and separating punctuation, they are one. But lowercasing a part number such as “A7” and “a7” would merge two different parts. The right normalisation depends on the data, and the choice should be recorded and tested.

Engineering implications and limitations

Normalise once, at one place. The pipeline that prepares training text and the pipeline that prepares prompts at inference must apply exactly the same steps. A mismatch (for example NFC in training, raw text in production) quietly degrades results.

Prefer reversible steps. If you can reconstruct the original text from the processed one, you keep the option to change your mind. Lowercasing and whitespace collapse are not reversible; a byte-level tokenizer on raw text is.

Treat whitespace as data for code and tables. Collapsing spaces destroys Python indentation and column alignment.

Deduplicate. Repeated passages are over-weighted in training, increase memorisation and can leak test data into training (Lesson LM2-T6). Exact deduplication is easy; near-duplicate detection needs hashing methods and is worth the effort at scale.

Know your licence and provenance. Record where each source came from and under what terms. Do not include data you may not use. This track uses small invented or openly licensed texts for its exercises.

Write down the preprocessing as code with tests. A test with a few ugly strings (accents, tabs, emoji, a stray HTML tag) catches silent changes when the pipeline is modified.

Robots. Voice and chat commands arrive with typos, casing and punctuation noise. Normalise for matching and logging, but keep the raw text to show to a user and for debugging.

Common misconceptions

Engineering takeaways

1. Each preprocessing step decides what counts as the same text, and removes some information to do so.
2. Unicode normalisation (usually NFC) makes equivalent characters identical: “café” has 4 or 5 code points depending on how it is written.
3. Type-token ratio TTR = V/N falls as a text grows; Zipf’s law f(r) ≈ C/r^s explains the long tail of rare words.
4. Modern GPT-style pipelines keep case, punctuation and whitespace, use a byte-level tokenizer and spend their effort on data selection and deduplication.
5. Apply identical preprocessing in training and in production, and keep the raw text.

References and further reading

The core reference is Chapter 2 of the book; the standards below define the normalisation forms used here.

• Unicode Consortium, Unicode Standard Annex #15: Unicode Normalization Forms defines NFC, NFD, NFKC and NFKD.
• Python documentation for the unicodedata module and collections.Counter.
• Build a Large Language Model (From Scratch), Chapter 2: Working with Text Data (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: how normalised text is cut into tokens and mapped to integer IDs: characters, words, subwords and the vocabulary.

Related content

Nothing below is required to finish this lesson.

• Before
  LM1-T5: What LLMs Can and Cannot Do, and the Build Plan

  State the capabilities and failure modes of LLMs honestly and map the stages of building a GPT-style model.

• Lab
  LM2-L1: Build a Simple Tokenizer

  Write encode and decode for a word-level vocabulary with special tokens and inspect the token IDs.