LM2-T5 — Input-Target Pairs and Sliding Windows

Learning objectives

After this lesson, you should be able to:

• Build input and target sequences by shifting a token sequence by one position.
• Compute the number of windows from the stream length, context length and stride.
• Explain how the stride controls overlap, redundancy and the effective number of training examples.
• Recognize off-by-one and leakage errors in a windowing function.

The idea: every position is a lesson

Take the token sequence t0 t1 t2 t3 t4. A model reading t0 t1 t2 t3 should predict t4. But it should also predict t1 from t0, t2 from t0 t1, and t3 from t0 t1 t2. If the model processes the whole window in one pass with a causal mask (Module 3), it makes all of these predictions at the same time. So we feed the input t0 t1 t2 t3 and the target t1 t2 t3 t4: the same sequence shifted left by one. One window of length L yields L training examples.

Context length

The context length L (also called block size or maximum sequence length) is the number of tokens the model reads at once. It is fixed by the architecture and is the longest history the model can use. The GPT-2 models in the book use 1,024 (the book’s own demos shorten it to 256 to fit on a laptop); our toy models use even less.

Stride

The stride s is how far the window moves between examples. With s = L, windows do not overlap: each token is an input exactly once. With s = 1, consecutive windows share all but one token, producing the maximum number of windows but with enormous redundancy. Smaller strides give more windows from the same text, but the extra windows are mostly the same data in a different position, so they do not add much information and cost proportionally more compute per epoch.

Why overlap is not free

Overlapping windows let the model see the same token at different positions in the context, a mild form of data augmentation. But training on the same text many times pushes the model toward memorizing it. For large corpora the usual choice is to cut non-overlapping windows or use a stride close to L, and rely on the sheer size of the data for variety. For tiny corpora, as in our exercises, overlap is a pragmatic way to get more batches, with the caveat that validation data must not overlap training data (Lesson LM2-T6).

The mathematics: how many windows, and what they look like

Let the stream have T tokens t₀, …, t_T−1. A window starting at position i gives

T

total number of tokens in the stream.

L

context length: tokens per input window; the target also has L tokens.

i

start position of the window; the last token used is t_i+L, so we need i + L ≤ T − 1.

x, y

input and target index sequences, each of shape (L,); stacking B of them gives tensors of shape (B, L).

Window starts are 0, s, 2s, … while i + L ≤ T − 1. The number of windows is

s

the stride: the shift between consecutive windows, an integer ≥ 1.

floor

rounds down: a partial window at the end is dropped.

n

the number of full input/target pairs; zero (or negative) if T ≤ L.

For T = 5,145 tokens, L = 256 and s = 256: (5,145 − 256 − 1) / 256 = 19.09, so n = 19 + 1 = 20 windows. For L = 4 and s = 1 on the same stream: (5,145 − 4 − 1) / 1 + 1 = 5,141 windows, about 257 times more windows that together cover 5,141 × 4 = 20,564 input positions, four times the stream length.

Redundancy. Each token (away from the ends) appears as an input in about L/s windows. Total input positions are n · L, so

With s = L each token is used once per epoch; with s = L/4, four times, four times the compute for little new information.

A worked example and a PyTorch dataset

Stream: 11 tokens with IDs 10 to 20, so [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. With L = 4 and s = 4: n = floor((11 − 4 − 1) / 4) + 1 = floor(1.5) + 1 = 2. Window 1: inputs [10, 11, 12, 13], targets [11, 12, 13, 14]. Window 2 starts at 4: inputs [14, 15, 16, 17], targets [15, 16, 17, 18]. The last tokens 19 and 20 are not used; with s = 1 they would be: n = 11 − 4 − 1 + 1 = 7 windows.

A dataset in PyTorch

import torch
from torch.utils.data import Dataset

class WindowDataset(Dataset):
    """Input/target windows over a token-ID stream (GPT-style next-token data)."""
    def __init__(self, ids, context_length, stride):
        self.x, self.y = [], []
        for i in range(0, len(ids) - context_length, stride):
            self.x.append(torch.tensor(ids[i : i + context_length]))
            self.y.append(torch.tensor(ids[i + 1 : i + context_length + 1]))
    def __len__(self):
        return len(self.x)
    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

ds = WindowDataset(list(range(10, 21)), context_length=4, stride=4)
print(len(ds))                    # 2
print(ds[0][0].tolist(), ds[0][1].tolist())   # [10, 11, 12, 13] [11, 12, 13, 14]
print(ds[1][0].tolist(), ds[1][1].tolist())   # [14, 15, 16, 17] [15, 16, 17, 18]

The loop bound len(ids) - context_length is the key line. It stops before the window whose target would run past the end of the stream. Replacing it by len(ids) produces a short final target or an index error.

Engineering implications and limitations

Check the shift. Write an assertion: for every window, x[1:] == y[:-1]. It catches off-by-one errors, which are by far the most common bug here and do not crash anything: a model trained on misaligned targets just learns badly.

Never let the window cross documents unintentionally. If documents are joined with separators, a window may include the end of one document and the start of another. That is acceptable and intended with the separator in place. Without it, the model learns that unrelated texts follow each other.

Drop or pad the tail deliberately. Stream pretraining usually drops the incomplete tail. Do not pad it with real tokens or reuse the start of the stream, which creates a false window.

Count, then train. Print n and compare it with the formula before a long run. A dataset with 12 windows used for “training” will overfit in seconds, and one with a million useless overlapping windows wastes hours.

Avoid leakage between splits. If training and validation windows overlap, the validation loss is measured on text the model has trained on, and looks much better than it should. Split the stream first, then window each part (Lesson LM2-T6).

Memory. The simple dataset stores every window as its own tensor. For a large corpus, store the token stream once and create windows on demand in __getitem__ instead.

For robots. Logged action or command sequences are streams too. If one logged run is a document, separate runs with a boundary token so that the model does not learn that the end of one mission leads to the start of another.

Common misconceptions

Engineering takeaways

1. Targets are the inputs shifted by one token; one window of length L yields L training predictions.
2. The number of windows is n = floor((T − L − 1)/s) + 1; for T = 5,145, L = 256, s = 256 it is 20.
3. The stride controls overlap: each token is an input about L/s times per epoch.
4. Assert that x[1:] equals y[:-1], check the loop bound, and count windows against the formula.
5. Split the stream before windowing so validation windows never overlap training windows.

References and further reading

See Chapter 2 of the book (data sampling with a sliding window) and the PyTorch documentation for torch.utils.data.

• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI technical report.
• Build a Large Language Model (From Scratch), Chapter 2: Working with Text Data (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: grouping windows into batches, shuffling, and splitting data so the validation loss means something.

Related content

Nothing below is required to finish this lesson.

• Before
  LM2-T4: Special Tokens and Unknown Text

  Use end-of-text, padding and unknown-token conventions, and see why byte-level tokenization never fails.

• Lab
  LM2-L3: Diagnose Sliding Windows and Batching

  Build input-target pairs and find the off-by-one, leakage and batching bugs in a data pipeline.