LM4-T1 — The GPT Architecture

Learning objectives

After this lesson, you should be able to:

• List the components of a decoder-only GPT in order and give the tensor shape between them.
• Explain what “decoder-only” means and how it differs from encoder and encoder-decoder models.
• Read a configuration (layers, width, heads, vocabulary, context length) and relate it to model size.
• Write a runnable skeleton of a GPT that shows how shapes flow.

The idea: read tokens, refine a vector per token, score the vocabulary

Follow one batch through the model. The integer IDs of shape (B, n) are looked up in the token embedding table and summed with position embeddings (Lessons LM3-T1 and LM3-T2), giving a tensor of shape (B, n, d): one vector of width d per token. Call it the residual stream. Each Transformer block reads the stream, computes a correction, and adds it back, so that the stream is refined step by step: early blocks tend to build local features, later ones more abstract ones. After N blocks, a final layer normalisation rescales the stream and a linear output head turns each vector into V numbers, one per vocabulary entry. These are the logits. A softmax over the logits of position i gives the model’s distribution for token i + 1.

Why “decoder-only”

The original Transformer had two stacks: an encoder that reads a source sentence in both directions, and a decoder that generates the translation one token at a time and may look back at the encoder’s output. A GPT keeps only the decoder-style stack, with the causal mask (Lesson LM3-T4) and without the encoder to look at. The input is the text so far, the output is the next token. That is enough for a model that continues text, and it turns out to be enough for question answering, summarisation and code if the task is written as text to be continued. Encoder-only models (such as BERT) read both directions and are trained to fill in blanks; they are good at classification and embedding but do not generate text naturally.

Same block, many times

All blocks have the same structure and the same shapes; only their learned weights differ. That is what makes the architecture easy to scale: more capacity comes from more blocks (depth) or wider vectors (d), and the code changes in two numbers of a configuration.

The GPT-2 family as anchors

The GPT-2 paper trained four sizes with the same design. Their configurations are worth remembering because they set the scale for the rest of the track.

Modern models are larger and have refinements (Module 10), but the skeleton is the same.

The mathematics: the model as a composition of functions

Write the token IDs as t (shape (B, n)), the token and position embedding tables as E and P, and the block function as F_l for block l. The forward pass is

E

token embedding table, shape (V, d); E[t] has shape (B, n, d).

P

position embedding table, shape (L, d), where L ≥ n is the context length; the first n rows are used and broadcast over the batch.

F_l

Transformer block l; it maps (B, n, d) to (B, n, d). Its weights are different in each block.

LN

layer normalisation (Lesson LM4-T2).

W_head

output matrix of shape (d, V); with weight sharing it is the transpose of E.

logits

shape (B, n, V): the unnormalised scores for the next token at every position.

The probability that the next token is v, given the tokens up to position i, is softmax(logits[b, i, :])_v. Training minimises the cross-entropy between this distribution and the actual next token at every position (Module 5).

Where the size comes from. The parameters are the embedding tables (V · d + L · d), the N blocks (each about 12d², derived in Lesson LM4-T5) and the head (V · d, or none when shared). For GPT-2 small, blocks account for 85 million of the 163 million parameters of the separate-head version and for 68% of the 124 million of the shared-head version.

Compute. A forward pass costs roughly 2 FLOPs per parameter per token (each weight takes part in one multiply and one add), plus the attention products that depend on the sequence length. So a 124-million-parameter model needs about 250 million FLOPs to process one token.

A worked example: following shapes through a skeleton

Take a small configuration: vocabulary V = 1,000, context L = 64, width d = 32, two blocks. Feed a batch of B = 3 sequences of n = 10 token IDs.

The code below builds the skeleton with nn.Identity in place of the blocks, to show the data flow before any block exists.

import torch
import torch.nn as nn

class GPTSkeleton(nn.Module):
    def __init__(self, vocab, context, d, nlayers):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab, d)
        self.pos_emb = nn.Embedding(context, d)
        self.blocks = nn.Sequential(*[nn.Identity() for _ in range(nlayers)])   # real blocks come in LM4-T4
        self.final_norm = nn.LayerNorm(d)
        self.out_head = nn.Linear(d, vocab, bias=False)

    def forward(self, idx):
        B, n = idx.shape
        x = self.tok_emb(idx) + self.pos_emb(torch.arange(n))
        x = self.blocks(x)
        return self.out_head(self.final_norm(x))

model = GPTSkeleton(vocab=1000, context=64, d=32, nlayers=2)
idx = torch.randint(0, 1000, (3, 10))
logits = model(idx)
print(tuple(idx.shape), "->", tuple(logits.shape))          # (3, 10) -> (3, 10, 1000)
print(sum(p.numel() for p in model.parameters()))           # 1000*32 + 64*32 + 2*32 + 32*1000 = 66112

The count printed is 66,112: the token table (32,000), the position table (2,048), the layer norm (64) and the head (32,000). The blocks contribute nothing yet because they are placeholders.

From logits to text

To generate, take the logits of the last position, convert them to a distribution, pick a token, append it to the input and repeat (Lesson LM1-T2, and Module 5). The model is run again on the longer sequence; Module 10 shows how to avoid recomputing the past.

Engineering implications and limitations

Configuration is the interface. Keep all sizes in one dictionary (vocabulary, context length, width, heads, layers, dropout, bias flags) and build the model from it. Every size-dependent number in your code should be derived from this dictionary.

Match the published numbers. When you reimplement a known model, check the parameter count against the published figure (Lab LM4-L1 and the notebook LM4-L3 do this for GPT-2 small). A mismatch of a few thousand parameters is a missing bias or norm.

Context length is built in. With learned position embeddings the model cannot take more than L tokens; generation code must crop the input to the last L tokens.

Depth versus width. Parameters grow linearly with depth and quadratically with width. Wider models parallelise better on hardware; deeper models are more sequential. Published scaling studies suggest that, at a fixed budget, both matter, but the details are outside this lesson.

Logits dominate activation memory in small models. The output tensor (B, n, V) is much larger than the residual stream when V ≫ d: for B = 8, n = 1,024, V = 50,257 it has 411 million entries, 1.6 GB in float32. Loss computation is therefore often done in chunks.

For robots. The same skeleton is used by vision-language-action models: images and robot states are turned into embeddings, and an action head replaces or supplements the vocabulary head (VLA track). The idea “a stack of blocks over a sequence of vectors” carries over unchanged.

Common misconceptions

Engineering takeaways

1. A GPT is token and position embeddings, N identical Transformer blocks, a final layer norm and a linear head that outputs logits of shape (B, n, V).
2. Decoder-only means a causal stack with no encoder; the task is always “continue the text”.
3. GPT-2 sizes: 12 blocks and d = 768 (124 M), 24 and 1,024 (355 M), 36 and 1,280 (774 M), 48 and 1,600 (1,558 M).
4. Every stage inside the stack keeps the shape (B, n, d); only the head changes the width to V.
5. A forward pass costs about 2 FLOPs per parameter per token, plus the attention products.

References and further reading

See Chapter 4 of the book (implementing a GPT model from scratch) for the architecture, the GPT-2 sized configuration and the skeleton built step by step.

• Vaswani, A. et al. (2017). Attention Is All You Need. arXiv:1706.03762.
• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI technical report; the four model sizes are described there.
• Build a Large Language Model (From Scratch), Chapter 4: Implementing a GPT Model from Scratch (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: layer normalisation and residual connections, which keep a deep stack trainable.

Related content

Nothing below is required to finish this lesson.

• Before
  LM3-T6: Tensor Shapes and Matrix Operations in Attention

  Track batch, sequence, head and feature dimensions through attention and debug shape errors.

• Lab
  LM4-L1: Count GPT Parameters and Tensor Shapes

  Write a parameter counter and shape tracer for a GPT configuration and check it against known sizes.