LM1-T4 — Parameters, Tokens and Context Windows

Learning objectives

After this lesson, you should be able to:

• Define parameters, training tokens and context window and say what each limits.
• Estimate the memory of a model from its parameter count and numeric precision.
• Estimate training compute from parameters and tokens, and use the tokens-per-parameter rule of thumb.
• Explain why context length is counted in tokens and what it costs.

The idea: size, data and memory

Parameters are the numbers the training process adjusts: the weights of every layer. A model with 124 million parameters has 124 million learned numbers. More parameters let the model represent more patterns, and cost more memory and arithmetic. They are the “size” in “model size”.

Training tokens are the amount of text the model sees during training (counting repeats if it reads the data more than once). More data means more patterns to learn from, and more time to train. A large model trained on too little data underperforms; a small one on a huge amount reaches its limits sooner.

The context window is the maximum number of tokens the model can take as input at once, prompt plus everything it has generated. Anything outside it is invisible (Lab LM1-L2 showed this with a toy model).

Why tokens and not words

Models count in tokens because tokens are what they read. A token is often a word or a piece of a word; for ordinary English text it is a rough rule that one token is about three quarters of a word, or four characters. Code, numbers and other languages may need more tokens per word. A 1,024-token window therefore holds roughly 770 English words, and a 128,000-token window about 96,000 words.

Bigger is not always better

Extra parameters help if there is data to use them and compute to train them. The research on scaling laws found that for a fixed compute budget there is a best balance between model size and data; training a very large model on too few tokens wastes compute (Hoffmann et al. 2022). In practice a smaller model trained well, or fine-tuned for a task, often beats a larger general model on that task, and it is cheaper to run. For a robot with a modest computer, size is a hard constraint, not a preference.

Where the parameters are

In a GPT-style model, the parameters sit in the token embedding table (vocabulary size times the embedding width), in the attention and feed-forward layers of every Transformer block, and in the normalisation layers. For GPT-2 small the embedding table alone is 50,257 × 768 = 38.6 million parameters, about a third of the total. Module 4 counts all of them exactly.

The mathematics: memory, compute and the data rule of thumb

Memory of the weights. With N_p parameters stored at b bytes each,

N_p

the number of parameters.

b

bytes per parameter: 4 for 32-bit floats, 2 for 16-bit floats, 1 for 8-bit integers, 0.5 for 4-bit.

This is a lower bound: running the model also needs activations and, for generation, a cache of past keys and values that grows with context length (Module 10).

Training compute. With D training tokens, a standard approximation for the total floating-point operations is

C

total operations to train, in FLOPs.

N_p, D

parameters and training tokens; the factor 6 combines about 2 operations per parameter per token for the forward pass and about 4 for the backward pass.

Tokens per parameter. The compute-optimal results of Hoffmann et al. (2022) are often summarised as a rule of thumb: train on roughly 20 tokens for each parameter. It is a guide to what is efficient for a given compute budget, not a law; models intended for cheap serving are deliberately trained on far more tokens than this.

Context and attention cost. In standard attention each token looks at every other token in the window, so the attention computation for a window of n tokens grows like n². Doubling the context roughly quadruples that part of the cost (Lesson LM3-T6 and Module 10).

A worked example: three models

Memory. The 7-billion-parameter model at 16 bits needs 7 × 10⁹ × 2 = 14 GB for the weights alone. At 8 bits it needs 7 GB and at 4 bits about 3.5 GB (Lesson LM8-T4), which is why quantised models fit on consumer hardware. GPT-3 at 16 bits needs 350 GB, more than a single accelerator holds.

Training compute. For GPT-3: 6 × 175 × 10⁹ × 300 × 10⁹ = 3.15 × 10²³ FLOPs, in line with the roughly 3 × 10²³ reported for training it. For a 124-million-parameter model on the 20-tokens-per-parameter rule, D = 20 × 124 × 10⁶ = 2.5 × 10⁹ tokens and C = 6 × 124 × 10⁶ × 2.5 × 10⁹ = 1.9 × 10¹⁸ FLOPs. At a sustained 10 TFLOP/s (a good single GPU in practice) that is 1.9 × 10¹⁸ / 10¹³ = 1.9 × 10⁵ s, a little over two days. At 100 GFLOP/s on a CPU it would take over 200 days. This is why the notebooks of this track pretrain a model with a few hundred thousand parameters, on a corpus of some tens of thousands of tokens: minutes, not months.

Context. A 2,048-token window holds roughly 1,500 words. A prompt that includes a 3,000-word document plus instructions needs about 4,000 tokens: too long for GPT-3’s window, but fine for a model with a window of 8,192 tokens. Doubling the window from 4,096 to 8,192 tokens makes the attention matrix 4 times larger.

A small calculator

def weight_memory_gb(nparams, bytes_per_param):
    return nparams * bytes_per_param / 1e9

def train_flops(nparams, ntokens):
    return 6 * nparams * ntokens

print(weight_memory_gb(7e9, 2))              # 14.0  (16-bit)
print(weight_memory_gb(7e9, 0.5))            # 3.5   (4-bit)
print(f"{train_flops(124e6, 20 * 124e6):.2e}")   # 1.85e+18

The functions only multiply. Their value is the habit: before running anything, estimate whether it fits in memory and finishes in a useful time.

Engineering implications and limitations

Estimate before you try. Parameters times bytes tells you whether a model fits in GPU memory; parameters times tokens tells you how long a training run will take. An hour of arithmetic saves days of failed runs.

Memory, not compute, usually decides on small hardware. A model that does not fit cannot run at all, however fast the processor. Quantisation and smaller models are the usual answers (Module 8).

Count tokens, not words, when you budget. Cost, speed and limits are all per token. Use the model’s own tokenizer to count (Module 2); rules of thumb vary with language and content.

A longer context is not free. It costs memory (the cache grows with length), time (the first token takes longer, Lesson LM10-T5) and often accuracy, because models can use information in the middle of a very long context less reliably than at the ends.

Match the model to the job and the machine. A robot that must answer in 200 ms on an embedded computer cannot use a 70-billion-parameter model locally. Plan for a smaller model, a remote service with a safe fallback, or a cached/structured approach.

Be honest about scale. The models you will build are millions of times smaller than production systems. They teach the mechanisms; they do not reproduce the capabilities that appear at scale.

Common misconceptions

Engineering takeaways

1. Parameters are learned numbers (memory = parameters × bytes per parameter); training tokens are the text seen in training; the context window is how many tokens the model can see at once.
2. Weight memory: 7 × 10⁹ parameters take 14 GB at 16-bit, 7 GB at 8-bit, about 3.5 GB at 4-bit.
3. Training compute is about 6 × parameters × tokens; a common compute-optimal guide is about 20 tokens per parameter.
4. Context is counted in tokens (about three quarters of an English word each); standard attention cost grows with the square of the context length.
5. Estimate memory and compute before running anything, and remember that the educational models here are tiny compared with production systems.

References and further reading

Chapter 1 of the book introduces model size and data; the papers are optional extensions on scaling.

• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556: the balance of model size and training tokens.
• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
• Brown, T. et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165: GPT-3 size, data and context.
• Hoffmann et al. (2022), Training Compute-Optimal Large Language Models (research paper)
• Build a Large Language Model (From Scratch), Chapter 1: Understanding Large Language Models (primary book reference)
• Companion site for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: what these models can and cannot do, and the plan for building one in the rest of the track.

Related content

Nothing below is required to finish this lesson.

• Before
  LM1-T3: Training Versus Inference

  Separate what happens when a model learns from what happens when it is used, and place pretraining and fine-tuning.

• Lab
  LM1-L2: Trace Text to Generated Output

  Follow a prompt through tokens, context, probabilities and a generated continuation, and see how context changes predictions.