LM3-T2 — Positional Information

Learning objectives

After this lesson, you should be able to:

• Explain why self-attention without position information is insensitive to token order.
• Add a learned absolute position embedding to token embeddings and state the shapes.
• Compute sinusoidal position encodings by hand from the formula.
• Compare absolute, sinusoidal and relative schemes by parameters, length limits and extrapolation.

The idea: attention is a set operation

Self-attention (Lesson LM3-T3) computes each output as a weighted average over all tokens, with weights that depend only on the content of the vectors. Nothing in the formula refers to a position. If you swap two inputs, their outputs swap too and nothing else changes. Formally the layer is permutation equivariant: permuting the input permutes the output the same way.

Recurrent networks received order for free, because they read one token at a time. A Transformer reads all tokens at once, which is what makes it fast to train, and pays for it by having to supply order explicitly.

Absolute position embeddings

The simplest fix: keep a second learned table with one row per position, 0 to L − 1 where L is the context length, and add the row for position i to the token embedding at position i. The model learns whatever a position vector must contain. This is what GPT-2 does, and what the book builds. Its limits: the table has only L rows, so the model cannot take longer inputs, and it learns no explicit relation between neighbouring positions; position 7 and 8 are just two different rows that happen to be used together often.

Sinusoidal encodings

The original Transformer used fixed (not learned) vectors built from sines and cosines of different frequencies. Position i gets a vector whose entries are sin(i/10000^2k/d) and cos(i/10000^2k/d) for k = 0, 1, … Fast-moving pairs distinguish adjacent positions; slow-moving pairs distinguish far-apart ones, like the digits of a clock. No parameters are needed, any length can be generated, and the dot product of two encodings depends only on their distance, which is convenient.

Relative schemes

What often matters is not “this is token 513” but “this token is three positions before that one”. Relative schemes put the offset directly into the attention scores. Rotary position embeddings (RoPE), used in many recent open models, rotate query and key vectors by an angle proportional to position, so that their dot product depends on the offset. These schemes extend more gracefully to long inputs. They are covered in Module 10; here it suffices to know that absolute learned embeddings are one design among several.

The mathematics: adding position vectors

Let the token embeddings of a sequence be the matrix X of shape (n, d) (Lesson LM3-T1). With a learned position table P of shape (L, d), the input to the first layer is

X

token embeddings, shape (n, d); the row i depends only on the token at position i.

P

position embeddings, shape (L, d); the row i depends only on the position i.

L

the context length: the longest sequence the model supports; n ≤ L.

Z

the sum, shape (n, d); the same shape as X, so no later layer changes.

For a batch of shape (B, n, d) the same (n, d) position matrix is added to every sequence by broadcasting. The position table adds L · d parameters: for GPT-2 small, 1,024 × 768 = 786,432.

Sinusoidal formula. For position i and pair index k = 0, …, d/2 − 1:

i

position in the sequence, 0, 1, 2, …

k

index of a sine/cosine pair; each pair has its own frequency ω_k = 1 / 10000^(2k/d).

d

the embedding width; even, so there are d/2 pairs.

PE[i, ·]

the d-dimensional position vector of position i; entries lie in [−1, 1].

With d = 4 there are two pairs, with ω₀ = 1 and ω₁ = 1/100. Then PE[0] = [0, 1, 0, 1]; PE[1] = [sin 1, cos 1, sin 0.01, cos 0.01] ≈ [0.841, 0.540, 0.010, 1.000]; PE[2] = [sin 2, cos 2, sin 0.02, cos 0.02] ≈ [0.909, −0.416, 0.020, 1.000]. The first pair turns quickly with position; the second barely moves in this short range. In a large model the frequencies span many orders of magnitude.

Distance property. By the identity cos(a − b) = cos a cos b + sin a sin b,

so two positions the same distance apart have the same dot product wherever they are. For d = 4 and m = 2: cos(2) + cos(0.02) ≈ −0.416 + 0.9998 = 0.584 for every i.

A worked example: order matters, and a learned table

Permutation test. Take two tokens with embeddings a and b. Self-attention on (a, b) gives outputs (o_a, o_b). On (b, a) it gives (o_b, o_a): the same two vectors in swapped order, so nothing in the outputs records which came first. The code below verifies this numerically, then breaks the symmetry by adding position vectors.

import torch

torch.manual_seed(0)
n, d = 3, 4
x = torch.randn(n, d)                       # three token embeddings
Wq, Wk, Wv = (torch.randn(d, d) for _ in range(3))

def self_attention(z):
    q, k, v = z @ Wq, z @ Wk, z @ Wv
    w = torch.softmax(q @ k.T / d ** 0.5, dim=-1)
    return w @ v

perm = torch.tensor([2, 0, 1])              # reorder the tokens
print(torch.allclose(self_attention(x)[perm], self_attention(x[perm]), atol=1e-5))   # True: outputs are just reordered

pos = torch.nn.Embedding(n, d)              # a learned position table with n rows (the book uses context_length rows)
p = pos(torch.arange(n))                    # shape (n, d): row i is the vector of position i
a = self_attention(x + p)
b = self_attention(x[perm] + p)
print(torch.allclose(a[perm], b, atol=1e-5))   # False: with positions the order changes the result

The first comparison is True: reordering inputs only reorders outputs. In the second, position vectors go with the slot, not the token, so putting the same tokens into different slots produces different sums, and the outputs genuinely differ.

Sinusoids in code

import math
import torch

def sinusoidal(n, d):
    pe = torch.zeros(n, d)
    pos = torch.arange(n).unsqueeze(1).float()                   # (n, 1)
    freq = 1.0 / 10000 ** (torch.arange(0, d, 2).float() / d)    # (d/2,)
    pe[:, 0::2] = torch.sin(pos * freq)
    pe[:, 1::2] = torch.cos(pos * freq)
    return pe

pe = sinusoidal(3, 4)
print(pe[1].tolist())                         # [0.841..., 0.540..., 0.0099998..., 0.99995...]
d1 = (pe[0] @ pe[2]).item()
d2 = (sinusoidal(12, 4)[5] @ sinusoidal(12, 4)[7]).item()
print(round(d1, 4), round(d2, 4))             # 0.5837 0.5837  (same offset, same dot product)

Engineering implications and limitations

Position tables set the context length. A learned table with L rows cannot represent position L. Feeding a longer sequence is an index error, so the tokenizer or the application must truncate or window (Lesson LM2-T5).

Extending the context is not free. You cannot just append rows to a trained table: the new rows are untrained. Relative schemes and some rescaling tricks allow longer inputs, but quality at lengths never seen in training needs checking (Module 10).

Add, do not concatenate. The usual choice is to add position to token embeddings so the width stays d. Concatenation would enlarge every later matrix.

Position and tokens share the vector. Because they are summed, the first layers must disentangle “what” from “where”. Models learn to do so, which is part of why a small d hurts.

Test with a shuffle. A cheap sanity test of a new implementation: shuffle the tokens of an example. If the model output on the last position is unchanged, position information is not reaching the layers.

Left versus right padding. With absolute positions the same text gets different position vectors if you pad on the left. Be consistent between training and inference, and pass correct position indices when batching (Module 9).

For robots. Logged sensor streams have their own notion of time: timestamps. Whether to use token position or real elapsed time as the “position” is a design decision for sequence models in robotics; the VLA track discusses how action sequences are positioned.

Common misconceptions

Engineering takeaways

1. Self-attention is permutation equivariant, so order must be supplied separately.
2. Absolute learned position embeddings: a table of shape (L, d) added to the token embeddings; GPT-2 small uses 1,024 × 768 = 786,432 parameters for them.
3. Sinusoidal encodings need no parameters and PE[i] · PE[i + m] depends only on the offset m.
4. Relative schemes such as RoPE (Module 10) put the offset in the attention scores and extend more gracefully to long inputs.
5. The position table sets the maximum context length of a learned-position model.

References and further reading

See Chapter 2 of the book (encoding word positions) for the absolute embedding used in this track; rotary embeddings are treated in Module 10.

• Vaswani, A. et al. (2017). Attention Is All You Need. arXiv:1706.03762 (sinusoidal encodings, Section 3.5).
• Build a Large Language Model (From Scratch), Chapter 3: Coding Attention Mechanisms (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: queries, keys and values, the three roles that turn embeddings into attention.

Related content

Nothing below is required to finish this lesson.

• Before
  LM3-T1: From Token IDs to Embedding Vectors

  Explain an embedding table as a learned lookup and compute similarities between embeddings.

• Lab
  LM3-L1: Compute Attention Step by Step

  Compute scores, softmax weights and context vectors for a short sentence by hand-written code.