LM7-T1 — Why Base Models Do Not Follow Instructions

Learning objectives

After this lesson, you should be able to:

• Explain why next-token pretraining produces a text continuer, not an assistant.
• List what instruction tuning changes: the data format, the part of the text that carries the loss, and the stop signal.
• Estimate the data and compute of an instruction-tuning run.
• State the limits: it elicits and shapes behaviour; it is not a reliable way to add knowledge.

The idea: the pattern “request, then answer” has to be taught

A pretrained model is trained to make real documents likely. In its data, a line that looks like a request is followed by whatever followed it in the document it came from: the next exercise, a comment thread, an answer on a forum, a blank line. It has seen all of these, and the next-token objective gives no reason to prefer one. Facing a request, it samples from this mixture. Often it continues the list, repeats the request or drifts to a related text.

What instruction tuning changes

• The data. Examples are written as instruction–response pairs inside a fixed template (Lesson LM7-T2). The model sees thousands of cases where the text following a request is a helpful, correctly formatted answer.
• The part that is learned. The loss is computed only on the response (Lesson LM7-T4). The prompt is context, not target.
• The stop signal. Every response ends with a special end-of-turn token (Lesson LM7-T3), so that the model learns when the answer is finished.

The architecture and the loss function are those of pretraining: next-token cross-entropy on tokens. The change is entirely in what text the model is shown and which positions count. That is why a few thousand examples can change behaviour so visibly: the model already has the abilities; instruction tuning selects which of its many possible behaviours to produce when it sees this template.

Where it fits in the pipeline

Assistant models are built in stages: pretraining (Module 5); supervised fine-tuning on demonstrations of good responses (this module); and often a preference stage in which people (or models) rank responses and the model is trained to prefer the better ones. The InstructGPT work reported that models trained this way, even with 1.3 billion parameters, were preferred by human evaluators to a 175-billion-parameter base model on the prompts tested, which shows how large the effect of this stage can be. Preference tuning is outside the scope of the book chapter and of this track beyond a mention in Module 9; supervised instruction tuning is the foundation it builds on.

What it does not do

Instruction tuning does not reliably teach new facts. A model tuned on examples containing facts it did not know learns to produce confident-sounding statements in the right style, which can increase fabrication. It teaches format, style, tone, task framing and refusal behaviour, and it can sharpen skills the model already has. New knowledge belongs in pretraining or in the prompt (retrieval, Module 11).

The mathematics: the same loss on selected tokens

Let a training sequence be a prompt x = (x₁, …, x_m) followed by a response y = (y₁, …, y_r) and an end token y_r+1 = ⟨end⟩. Pretraining would minimise the loss over every position of the concatenation. Instruction tuning minimises

x

the prompt: instruction and input, formatted by the template; it conditions the prediction but is not predicted.

y_j

the j-th response token; y_r+1 is the end-of-turn token.

θ

all model parameters, initialised from the pretrained model.

r + 1

the number of positions that count: response tokens plus the end token.

Compared with the pretraining loss, the only change is the set of positions summed over. This is why the same training code serves both stages once the labels are masked.

Scale. Instruction sets range from about a thousand carefully written examples to hundreds of thousands. A set of 1,100 examples averaging 100 tokens, trained for 2 epochs, processes 1,100 × 100 × 2 = 220,000 tokens. For a 355-million-parameter model that costs 6 · 3.55·10⁸ · 2.2·10⁵ ≈ 4.7·10¹⁴ FLOPs: minutes on one GPU. Compare pretraining the same model on 7·10⁹ tokens (20 tokens per parameter): 1.5·10¹⁹ FLOPs, about 32,000 times more.

Quality over quantity. Studies of instruction tuning (for example the LIMA paper, which used a thousand curated examples) suggest that a small number of diverse, high-quality demonstrations can be enough to elicit good behaviour from a strong base model, while large amounts of noisy data may not help. The base model matters more than the instruction set; the instruction set shapes the style.

A worked example: what counts and what it costs

A single example in the template of Lesson LM7-T2 has a 12-token prompt, a 6-token response and one end token. Of its 19 tokens, 7 carry the loss (37%). In a batch with 4 padding tokens per example, 7 of 23 positions count (30%). If the whole sequence were in the loss, 70% of the gradient signal would go to predicting text that the user supplies.

flops = lambda params, tokens: 6 * params * tokens

sft_tokens = 1100 * 100 * 2                  # examples x tokens each x epochs
pre_tokens = 20 * 355_000_000                # 20 tokens per parameter
p = 355_000_000
print(sft_tokens, f"{flops(p, sft_tokens):.1e}", f"{flops(p, pre_tokens):.1e}")   # 220000 4.7e+14 1.5e+19
print(round(flops(p, pre_tokens) / flops(p, sft_tokens)))                          # 32273

prompt, response, end, pad = 12, 6, 1, 4
total = prompt + response + end + pad
print(total, response + end, round((response + end) / total, 2))                  # 23 7 0.3

# mean loss with and without masking, for assumed per-token losses
loss = {"prompt": 3.0, "response": 0.8, "end": 2.6, "pad": 5.0}
all_tokens = (prompt * loss["prompt"] + response * loss["response"] + end * loss["end"] + pad * loss["pad"]) / total
masked = (response * loss["response"] + end * loss["end"]) / (response + end)
print(round(all_tokens, 3), round(masked, 3))                                      # 2.757 1.057

The two printed means (2.757 and 1.057) show how much of an unmasked loss is not about the response. The unmasked number is also higher for a reason that has nothing to do with answer quality, so the loss curve of an unmasked run cannot be read as progress at the task.

Why a base model can still be prompted

Base models can often be steered by showing a pattern: a few examples of request and answer in the prompt (few-shot prompting) make the answer the most likely continuation. Instruction tuning builds that pattern into the weights, so that a zero-shot request in the template works without examples and the model stops cleanly. Few-shot prompting and instruction tuning address the same gap in different ways (Module 11 returns to prompting).

Engineering implications and limitations

Start from the best base model you can run. Instruction tuning elicits abilities; it does not create them. A small base model tuned on excellent data is still a small model.

Keep the template fixed. The model is tuned to the exact format. Use it at inference, with the response section left open, as in training (Lesson LM7-T2).

Do not expect new knowledge. If the model must answer questions about your robot’s manuals, give it the relevant text in the prompt (retrieval) rather than hoping to tune the facts in.

Watch for loss of ability. Training on a narrow set can reduce general performance. Evaluate on a few unrelated tasks before and after, and prefer small learning rates and few epochs (Lesson LM6-T1, Module 8).

Safety behaviour is also data. Whether a model refuses unsafe requests is largely determined by its instruction and preference data. Include examples of the behaviour you want, including refusals, and test it (Module 9).

Use the license terms. Instruction data generated by another provider’s model may be restricted by that provider’s terms; check before training on it or releasing the result.

For robots. An assistant that turns operator requests into actions needs the end-of-turn token and a rigid output format more than eloquence. Tune on the exact request and action formats, and validate outputs with a schema (Module 11).

Common misconceptions

Engineering takeaways

1. A base model continues text; nothing in pretraining prefers answering a request over continuing it.
2. Instruction tuning is next-token training with three changes: instruction–response data in a template, loss only on response and end token, and an end-of-turn stop signal.
3. A run of 1,100 examples × 100 tokens × 2 epochs on a 355 M model costs about 4.7e14 FLOPs, some 32,000 times less than pretraining it.
4. It elicits and shapes behaviour well, and teaches facts poorly; give knowledge through pretraining or retrieval.
5. Data quality and diversity matter more than volume, and the base model sets the ceiling.

References and further reading

See Chapter 7 of the book (finetuning to follow instructions: introduction to instruction finetuning).

• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
• Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. arXiv:2305.11206.
• Build a Large Language Model (From Scratch), Chapter 7: Finetuning to Follow Instructions (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: how instruction datasets are built and formatted with a prompt template.

Related content

Nothing below is required to finish this lesson.

• Before
  LM6-T5: Generalization and Failure Analysis

  Inspect errors, spot label noise and shortcuts, and judge whether a classifier will transfer.

• Lab
  LM7-L1: Format Instruction Data with Templates

  Write a prompt template and tokenizer-side formatter, and find malformed examples.