LM7-T2 — Instruction Datasets and Prompt Formats

Learning objectives

After this lesson, you should be able to:

• Describe the fields of an instruction example and the common sources of such data.
• Format a record into a prompt and a full training text with a template, handling empty inputs.
• Estimate the token overhead of a template and the length distribution of a set.
• Run basic cleaning checks: empty fields, marker collisions, duplicates and conflicts.

The idea: show the model the job, many times, in one fixed shape

An instruction example has an instruction (what to do), an optional input (the material to do it on) and a response (a good answer). “Convert to metres” with the input “250 centimetres” and the response “2.5 metres” is one example; “Say hello” with no input is another. The model needs thousands of such records, across many kinds of task, to learn the general pattern.

Where the data come from

• Written by people. Highest quality and most expensive. Annotators write instructions and ideal responses, or domain experts write them for specialist tasks.
• Generated by a stronger model. Cheap and large (the Self-Instruct approach bootstraps instructions from a small seed set). Quality depends on the generator; errors and style are inherited; licences may restrict use.
• Converted from existing resources. Question–answer sets, classification data written as instructions, logs of real interactions (with consent and privacy handling).
• Mixtures. Most practical sets combine sources and weight them.

What makes a set good

Diversity of tasks, lengths and phrasings; correctness of responses; consistency in style and format; coverage of the behaviours you need, including refusals, clarifying questions and “I don’t know”; and no leakage of the test set. Studies suggest that a small, carefully curated set can match much larger noisy ones. Thousands of near-identical examples teach one thing very thoroughly and others not at all.

The template

The template turns the fields into text with markers that make the three parts recoverable and a final end-of-turn token. A common design, used in the book and in this track, has labelled sections: an instruction header, an input header present only when there is an input, and a response header. Everything up to and including the response header is the prompt; the response and the end token are what the model learns to produce. The same template must be used for inference, or the model faces a format it was not tuned on.

The mathematics: tokens, overhead and length

Let a record have an instruction of a tokens, an input of b tokens (possibly 0) and a response of r tokens. The template adds fixed overhead tokens: the section headers and separators, h₀ without input and h₁ with input, plus one end token. The length of a training sequence is

a, b, r

token counts of instruction, input and response.

h(b)

template overhead: h₀ tokens when the input is empty (the Input section is omitted), h₁ when present.

n

training sequence length in tokens, including the end token; it must not exceed the context length.

With the character-level template of the notebook, h₀ and h₁ are 33 and 46 characters. With a subword tokenizer the three headers cost roughly 15 to 25 tokens. For short examples, such as “Say hello” → “Hello” (8 tokens of content), the template dominates the length: overhead is 70% or more of the sequence. This is acceptable because the headers are easy to predict (low loss) and are masked anyway (Lesson LM7-T4), but it matters for compute and context budget.

Length distribution. Summarise the distribution of n: the mean, the 95th percentile and the maximum. Choose the training context length at or above the 95th percentile and decide what to do with longer examples (truncate the end of long responses only with care, or drop them). Batches padded to the longest example waste compute in proportion to the spread of lengths; group similar lengths together (bucketing) to reduce it.

Dataset size and epochs. For N examples of average length n̄, one epoch processes N · n̄ tokens. A set of 1,100 examples at 100 tokens is 110,000 tokens per epoch. Two or three epochs are usual for small sets; many more invite memorisation (Lesson LM5-T4).

A worked example: formatter and cleaning checks

The code below is a compact version of what Lab LM7-L1 asks you to write: the formatter with the optional Input section, and four data checks. Run it to see the checks flag the injected faults.

END = "<|end|>"

def format_example(e):
    prompt = f"### Instruction:\n{e['instruction']}\n\n"
    if e["input"]:
        prompt += f"### Input:\n{e['input']}\n\n"
    prompt += "### Response:\n"
    return prompt, prompt + e["response"] + END

data = [
    {"instruction": "Convert to metres", "input": "250 centimetres", "response": "2.5 metres"},
    {"instruction": "Say hello", "input": "", "response": "Hello, operator."},
    {"instruction": "  ", "input": "", "response": "Hello"},                                  # empty instruction
    {"instruction": "Name the colour", "input": "red box", "response": ""},                    # empty response
    {"instruction": "Echo the text", "input": "x", "response": "### Response: again"},         # marker inside the text
    {"instruction": "Convert to metres", "input": "250 centimetres", "response": "2.50 m"},    # conflicts with the first
]

prompt, full = format_example(data[1])
print(repr(prompt))      # '### Instruction:\nSay hello\n\n### Response:\n'
print(repr(full))        # '### Instruction:\nSay hello\n\n### Response:\nHello, operator.<|end|>'

markers = (END, "### Instruction:", "### Input:", "### Response:")
seen, problems = {}, []
for i, e in enumerate(data):
    if not e["instruction"].strip():
        problems.append((i, "empty_instruction"))
    elif not e["response"].strip():
        problems.append((i, "empty_response"))
    elif any(m in e[k] for m in markers for k in ("instruction", "input", "response")):
        problems.append((i, "marker_in_text"))
    else:
        key = (e["instruction"], e["input"])
        if key in seen and seen[key] != e["response"]:
            problems.append((i, "conflict"))
        seen.setdefault(key, e["response"])
print(problems)          # [(2, 'empty_instruction'), (3, 'empty_response'), (4, 'marker_in_text'), (5, 'conflict')]

lengths = sorted(len(format_example(e)[1]) for e in data)
print(lengths, "median", lengths[len(lengths) // 2])

Each check is cheap and each catches a failure that would otherwise silently shape the model. Beyond them, spot-read a random sample of fifty records: no program finds a wrong answer that looks well formed.

Choosing the format

Engineering implications and limitations

Version the data. Store the data, the template and the cleaning script together under version control. A model trained on “instruction data v3 with template B” is reproducible; one trained on “some instructions” is not.

Split before cleaning decisions. Hold out an evaluation set first and keep it out of every later step, including near-duplicate removal against the training data (Lesson LM7-T5).

Deduplicate. Exact duplicates by hash, near-duplicates by normalised text or n-gram overlap. Repeated examples are memorised and can dominate small sets.

Remove private data. Scan for personal information, credentials and internal identifiers; instruction data from logs often contain them. A model can reproduce what it has seen.

Check licences. For each source: who owns it, what does the licence allow, do terms restrict training or competing models? Record the answer with the data.

Balance task types. Count examples per category and per length bucket. A set that is 70% one task will tune the model towards it.

Include the behaviours you want at the edges. Examples of correctly saying “I cannot do that”, asking for missing information, and ignoring instructions inside the input text that should be treated as data.

For robots. Write instruction data in the exact format of the command interface (units, identifiers, allowed actions), include negative examples for unsafe or impossible requests, and keep a set of real operator phrasings for evaluation.

Common misconceptions

Engineering takeaways

1. An instruction example has an instruction, an optional input and a response; the template turns it into a prompt and a full text ending in an end-of-turn token.
2. Sources: human-written (best, costly), model-generated (cheap, inherits errors and licences), converted resources, mixtures.
3. Diversity, correctness, consistency and coverage matter more than raw count.
4. Template overhead is large for short examples; measure the length distribution and choose the context length from its 95th percentile.
5. Automate cheap checks (empty fields, marker collisions, duplicates and conflicts), spot-read a sample, hold out evaluation data first.

References and further reading

See Chapter 7 of the book (preparing a dataset for supervised instruction finetuning, and organizing data into training batches), where a prompt style with instruction, input and response sections is used.

• Wang, Y. et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560.
• Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. arXiv:2305.11206.
• Build a Large Language Model (From Scratch), Chapter 7: Finetuning to Follow Instructions (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: special tokens and chat templates, which mark roles and the end of each turn.

Related content

Nothing below is required to finish this lesson.

• Before
  LM7-T1: Why Base Models Do Not Follow Instructions

  Show that next-token pretraining continues text rather than answering it, and what instruction tuning changes.

• Lab
  LM7-L1: Format Instruction Data with Templates

  Write a prompt template and tokenizer-side formatter, and find malformed examples.