LM10-T1 — Decoder-Only Models Then and Now

Learning objectives

After this lesson, you should be able to:

• List the four architectural changes between a minimal GPT block and a Llama-style block.
• Count the parameters of both blocks from d, the number of layers, the feed-forward width and the vocabulary.
• State what did not change: the objective, the causal mask, residual streams and the training recipe.
• Read a model configuration and predict its size, its cache per token and its features.

The idea: same skeleton, better parts

A decoder-only transformer is a stack of identical blocks between an embedding table and an output projection. Each block lets every position look at earlier positions (attention) and then transforms each position separately (feed-forward). That skeleton has been stable since the 2018 and 2019 GPT models. What changed afterwards is not the idea but the parts, and almost every change was driven by one of three costs: training stability, quality for a fixed budget of parameters, or the cost of generating text.

Change 1: positions are rotated, not added

The minimal GPT adds a learned vector for each position to the token embedding once, at the input. Modern models usually leave the embeddings alone and instead rotate the query and key vectors inside every attention layer by an angle that depends on the position. The score between two tokens then depends on how far apart they are rather than on where they are in the window. This is rotary position embedding (RoPE, Lesson LM10-T2). It adds no parameters and extends more gracefully to longer contexts.

Change 2: keys and values are shared

In multi-head attention each of the H query heads has its own key head and value head. During generation the keys and values of all past tokens are stored (the KV cache, Lesson LM10-T5), and that store is often the largest consumer of memory. Grouped-query attention (GQA, Lesson LM10-T3) lets groups of query heads share one key-value head, cutting the cache by the group size while keeping the number of query heads.

Change 3: simpler normalisation, gated feed-forward layers

Layer normalisation subtracts the mean and divides by the standard deviation; RMSNorm only divides by the root mean square, which is slightly cheaper and works as well in practice. The feed-forward layer of Module 4 is two matrices with a GELU between them; modern models use a gated layer (SwiGLU) with three matrices, in which one projection of the input decides, element by element, how much of another projection passes through (Lesson LM10-T4).

Change 4: sometimes, only part of the feed-forward capacity runs

A mixture of experts replaces the one feed-forward layer by many, with a small router that sends each token to a few of them. The model stores many more parameters than it uses for any one token (Lesson LM10-T4).

What did not change

The objective is still next-token prediction with cross-entropy. The causal mask is the same. The residual stream is still the backbone. Pretraining still uses AdamW with warm-up and decay, the instruction-tuning and evaluation methods of Modules 7 and 9 apply unchanged, and LoRA (Module 8) attaches to the same projection matrices. If you understand the Module 4 model, you understand 90% of a modern one.

Two kinds of change

The mathematics: counting a block from its configuration

Let d be the model width, H the number of query heads of width d_h = d/H, H_kv the number of key-value heads, h the feed-forward hidden width, V the vocabulary and L the number of layers.

Minimal GPT block (with biases)

For GPT-2 small (d = 768, L = 12): a block has 7,087,872 parameters; with the token table (V = 50,257), the position table (1,024) and the final norm, and the output projection tied to the token table, the model has 124,439,808 parameters, the figure derived in Module 4.

Modern block (no biases, grouped-query attention, SwiGLU)

d², d²

the query and output projections, unchanged by sharing.

2·d·H_kv·d_h

the key and value projections; with H_kv = H this is 2d².

3·d·h

gate (d × h), up (d × h) and down (h × d); h is about 8d/3 to keep the size near 8d².

2·d

one learned scale vector per RMSNorm; there is no bias or shift.

A 7-billion-parameter configuration. d = 4,096, L = 32, H = H_kv = 32, d_h = 128, h = 11,008, V = 32,000, embeddings and output projection not tied. Attention: 4 · 4,096² = 67,108,864. Feed-forward: 3 · 4,096 · 11,008 = 135,266,304. Norms: 8,192. One block: 67,108,864 + 135,266,304 + 8,192 = 202,383,360. Thirty-two blocks: 6,476,267,520. Embedding table and output projection: 2 · 32,000 · 4,096 = 262,144,000. Final norm: 4,096. Total 6,738,415,616, the size of the 7-billion-parameter model of the Llama 2 family.

Two features of this count deserve a remark. First, the feed-forward layers hold about two thirds of the block’s parameters (135 of 202 million) in both generations of model. Second, the embedding tables are a small part (4%) of a large model but a big part (31%) of GPT-2 small, which is why small models often tie input and output embeddings and large ones often do not.

Cache per token

With keys and values stored in b bytes, the cache per token is 2 · L · H_kv · d_h · b. GPT-2 small: 2 · 12 · 12 · 64 · 2 = 36,864 bytes in 16 bits. The 7B configuration above: 2 · 32 · 32 · 128 · 2 = 524,288 bytes, half a megabyte per token; with 8 key-value heads it would be 131,072.

A worked example: read a configuration

The function below turns a dictionary of hyperparameters into the parameter count, the cache per token and a one-line summary. Run it on the two configurations of this lesson, then on a third that you invent to see how the choices move the numbers.

def describe(c):
    d, L, V, H = c['d'], c['layers'], c['vocab'], c['heads']
    hd = d // H
    kv = c.get('kv_heads', H)
    if c['ffn'] == 'gelu':                        # two matrices, width 4d, with biases
        h = 4 * d
        ffn = 2 * d * h + h + d
    else:                                         # SwiGLU: gate, up, down, no biases
        h = c['hidden']
        ffn = 3 * d * h
    bias = 4 * d if c['bias'] else 0
    attn = d * d + 2 * d * kv * hd + d * d + bias
    norms = 2 * (2 * d if c['norm'] == 'layer' else d)
    block = attn + ffn + norms
    tables = V * d * (1 if c['tied'] else 2) + c.get('positions', 0) * d
    final_norm = 2 * d if c['norm'] == 'layer' else d
    total = L * block + tables + final_norm
    cache = 2 * L * kv * hd * 2                   # 16-bit keys and values
    return {'block': block, 'total': total, 'cache_bytes_per_token': cache, 'ffn_share': round(ffn / block, 3)}

gpt2 = dict(d=768, layers=12, vocab=50257, heads=12, ffn='gelu', bias=True, norm='layer', tied=True, positions=1024)
big = dict(d=4096, layers=32, vocab=32000, heads=32, ffn='swiglu', hidden=11008, bias=False, norm='rms', tied=False)
print(describe(gpt2))    # {'block': 7087872, 'total': 124439808, 'cache_bytes_per_token': 36864, 'ffn_share': 0.666}
print(describe(big))     # {'block': 202383360, 'total': 6738415616, 'cache_bytes_per_token': 524288, 'ffn_share': 0.668}
print(describe(dict(big, kv_heads=8))['cache_bytes_per_token'])   # 131072 with grouped-query attention
print(describe(big)['total'] - describe(dict(big, kv_heads=8))['total'])   # 805306368

The last line prints the number of parameters removed by sharing key-value heads, 805,306,368; check it by hand: per layer the key and value projections shrink from 2 · 4,096² = 33,554,432 to 2 · 4,096 · 1,024 = 8,388,608, a saving of 25,165,824, times 32 layers = 805,306,368 parameters, about 12% of the model (805,306,368 / 6,738,415,616). The memory saving in the cache is 75%; the saving in parameters is 12%. That asymmetry is the whole point of the technique.

A reading guide for configurations

Engineering implications and limitations

Check the architecture before the weights. A checkpoint is useless to your code unless you implement exactly its block: the position scheme, the normalisation, the activation, the number of key-value heads, whether biases exist, whether embeddings are tied. Most conversion bugs are mismatches of exactly these.

Reproduce the parameter count. If your implementation of a published configuration does not reproduce the published parameter count to the last digit, something differs. This is the cheapest correctness test you have.

Choose per-token cost deliberately. Two models with the same parameter count can differ by 4× in cache per token. For long-context or high-concurrency serving the cache, not the parameter count, decides which model you can host.

Do not copy features without their reason. Grouped-query attention matters for serving; for a small model that you run on a single sequence it saves little. Rotary embeddings help length generalisation; for a short fixed context learned positions work fine.

Keep your Module 4 tests. The leakage test, the shape checks and the parameter-count test carry over to the modern block with only the formulas changed.

For robots. On-board models are small and memory-limited, so the cache and the weight precision matter more than the headline size. Read the configuration and compute both before you commit to a model for a robot; link the result to the memory and power budget of the platform (see the Physical AI track for the robot side).

Common misconceptions

Engineering takeaways

1. A modern open model is a decoder-only transformer with four changed parts: rotary positions, grouped-query attention, RMSNorm with gated feed-forward layers, and sometimes experts.
2. The objective, the causal mask, the residual stream and the training and fine-tuning methods are unchanged.
3. Block parameters: attention d² + 2·d·H_kv·d_h + d², feed-forward 3·d·h (gated), norms 2d. The 7B configuration has 6,738,415,616 parameters.
4. The feed-forward layers hold about two thirds of the block; sharing key-value heads removes about 12% of the parameters but 75% of the cache in the example.
5. Read a configuration before using weights, and reproduce its parameter count.

References and further reading

Module 4 of this track derives the 124,439,808 parameters of the small GPT; the Llama 2 report states the architecture used for the 7B count here. The book’s later chapters and the companion repository show equivalent implementations of these parts. All counts were computed by the code in this lesson.

• Radford, A. et al. (2019). Language models are unsupervised multitask learners (the GPT-2 report).
• Touvron, H. et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
• Su, J. et al. (2021). RoFormer: Enhanced transformer with rotary position embedding. arXiv:2104.09864.
• Ainslie, J. et al. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv:2305.13245.
• Shazeer, N. (2020). GLU variants improve transformer. arXiv:2002.05202.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: rotary position embeddings, the way modern models tell attention where tokens are.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T6: Responsible Evaluation and Model Limitations

  Report limits honestly, test failure cases and avoid conclusions from a few impressive examples.

• Lab
  LM10-L2: Compare Attention Variants

  Count parameters, cache size and compute for multi-head, multi-query and grouped-query attention.