LM12-T1 — Capstone Brief and Scoping for Limited Compute

Learning objectives

After this lesson, you should be able to:

• Choose a capstone project that fits CPU or modest GPU compute and state a measurable success criterion.
• Estimate parameters, training FLOPs, time and memory from simple formulas.
• Plan four milestones with decision gates and the baselines that each result must beat.
• Write a one-page brief and a risk list.

The idea: show the whole method on something small

What is assessed in a capstone is the method, not the size: a clear question, honest data handling, a baseline, a controlled experiment, an evaluation with uncertainty, an application that treats the model as untrusted, and a report whose claims match its evidence. A tiny model on a closed task shows all of that. A 7-billion-parameter fine-tune that you cannot afford to repeat shows none of it, because you cannot run the baselines, the ablation or the extra seeds.

Pick a project type that fits

If you are unsure, choose A or D: both run on a laptop and leave room for baselines, seeds and failure analysis. Whatever you choose, state the question as something with a measurable answer: “Does masking the prompt in the loss improve exact match on unseen bay numbers?” is a question; “build a good model” is not.

The one-page brief

1. Question,One sentence, answerable by an experiment.
2. Data,Source, size, licence, how it is split, what is held out and why.
3. Baselines,At least two: a trivial one (majority, nearest neighbour, random) and a competent one (a rule-based system, a simpler model, a pretrained model you do not train).
4. Method,The model and training setup in three lines; what you vary in the ablation.
5. Metrics and success criterion,The metric, the test set, the interval method and the threshold that counts as success, fixed before you see results.
6. Compute budget,Estimated FLOPs, memory and wall-clock, and what you will cut if it is too much.
7. Risks,The three things most likely to go wrong, each with a mitigation.

Write the success criterion first

The mathematics: what will it cost?

Four formulas cover most scoping decisions. For a model with N parameters trained on D tokens (Module 5):

N

number of parameters; for a GPT with width d, L layers, vocabulary V and context n_ctx: L(12d² + 13d) + 2Vd + n_ctx·d + 2d (for the PyTorch attention module with biases).

D

tokens processed = batch size × sequence length × steps.

effective FLOP/s

what the hardware really delivers on your code, usually 20 to 60% of the peak on a GPU and much less on a CPU for small models; measure it once.

The default project. The tiny GPT of notebooks LM12-L2 and LM12-L3 has d = 64, L = 2, V = 49, n_ctx = 75: N = 111,168 (the code below, and PyTorch, agree). A run of 1,000 steps with batch 32 and sequence 75 processes D = 2.4 million tokens: 6 · 111,168 · 2.4·10⁶ = 1.6·10¹² FLOPs. At an effective 50 GFLOP/s this is 32 s, which matches the 20 to 25 seconds measured on a laptop to within the accuracy of the assumption. Training memory is 16 · 111,168 = 1.78 MB plus activations: trivially small.

A larger project. One pass of a 124-million-parameter model over one million tokens costs 6 · 124·10⁶ · 10⁶ = 7.4·10¹⁴ FLOPs, 4.1 hours at 50 GFLOP/s, and 16 · 124·10⁶ = 2 GB of training state before activations. Pretraining it on 300 million tokens would take 1,240 hours on that CPU and about 1.6 hours on a 100 TFLOP/s GPU at 40% utilisation: possible there, not on a laptop. Fine-tuning for classification on 20,000 tokens: 0.08 hours on the CPU.

Repeats multiply cost. Baselines, an ablation with two variants and three seeds each means 6 training runs: budget six times the single-run time, plus evaluation. A project that can be run once can be reported only as an anecdote.

A worked example: the budget calculator

The code computes parameters, FLOPs, time and memory for the default project and for two larger ones, and prints a comparison across hardware. Use it for your own project before you commit to it.

def gpt_params(d, layers, vocab, ctx):
    block = 12 * d * d + 13 * d                  # attention (4d² + 4d), MLP (8d² + 5d), two LayerNorms (4d)
    return layers * block + vocab * d + ctx * d + 2 * d + vocab * d     # blocks, token table, positions, final norm, output head

def train_flops(params, batch, seq, steps):
    return 6 * params * batch * seq * steps      # about 6 FLOPs per parameter per token (forward + backward)

def train_memory_bytes(params):
    return 16 * params                           # float32 weights, gradients and two AdamW moments

def hours(flops, flops_per_second):
    return flops / flops_per_second / 3600

n = gpt_params(64, 2, 49, 75)
print(n)                                         # 111168
f = train_flops(n, 32, 75, 1000)
print(f"{f:.3g}", round(f / 5e10, 1))            # 1.6e+12 32.0   (seconds at an effective 50 GFLOP/s)
print(round(train_memory_bytes(n) / 1e6, 2))     # 1.78  (MB)

big = train_flops(124e6, 1, 1_000_000, 1)        # one pass over one million tokens with a 124M-parameter model
print(f"{big:.3g}", round(hours(big, 5e10), 1))  # 7.44e+14 4.1  (hours on the same CPU)
print(round(train_memory_bytes(124e6) / 1e9, 2)) # 1.98  (GB, before activations)

options = [("command parser from scratch", 1.6e12), ("classifier fine-tune, 124M, 20k tokens", 6 * 124e6 * 20_000),
           ("pretrain 124M on 300M tokens", 6 * 124e6 * 300e6)]
for name, fl in options:
    print(f"{name:42s} {hours(fl, 5e10):10.2f} h on a CPU, {hours(fl, 1e14 * 0.4):8.3f} h on a 100 TFLOP/s GPU at 40%")
# command parser from scratch                      0.01 h on a CPU,    0.000 h on a 100 TFLOP/s GPU at 40%
# classifier fine-tune, 124M, 20k tokens           0.08 h on a CPU,    0.000 h on a 100 TFLOP/s GPU at 40%
# pretrain 124M on 300M tokens                  1240.00 h on a CPU,    1.550 h on a 100 TFLOP/s GPU at 40%

The effective rate of 50 GFLOP/s is an assumption; if your first run takes twice as long as predicted, halve the rate and recompute. The point of the calculator is to be right about the order of magnitude: a project that needs 1,240 hours does not fit a laptop however carefully you plan the milestones.

The four milestones and their gates

Risk list for the default project

Engineering implications and limitations

Do the arithmetic before the code. A five-line calculator prevents most infeasible plans. Record the prediction, then compare it with the measured time and learn your effective FLOP/s.

Keep a project repository from day one. One folder, one environment file, one script per milestone, results written to files, and the brief at the top of the README. You will thank yourself at milestone 4.

Fix the evaluation before the model. Build the test set and the metric in milestone 1 and do not look at them while developing. The temptation to change the test set to make a result work is the largest source of unreproducible results.

Reserve a third of the time for evaluation and writing. Training is the visible part; the evidence and the report are what is judged.

Prefer a smaller question answered well. A clean negative result (“masking made no detectable difference with this setup and these seeds”) is publishable in a report; a muddled positive one is not.

For robots. If your project drives or advises a robot, the application layer and the safety policy are part of the scope, not an extension. Decide in the brief which actions the system may ever propose, and test the validator with hostile input (Module 11). The Physical AI and VLA tracks on this site cover the robot side in depth.

Common misconceptions

Engineering takeaways

1. Choose the smallest project that shows the whole method, state a measurable question and fix the success criterion before running anything.
2. Training cost: FLOPs ≈ 6·N·D, memory ≈ 16·N bytes plus activations; time = FLOPs / effective FLOP/s. The default project has 111,168 parameters, 1.6·10¹² FLOPs per run and takes about half a minute on a laptop.
3. Plan four milestones with three gates (fit the compute, beat the baselines, back every claim) and cut scope at a gate rather than late.
4. Budget for repeats: baselines, ablation variants and several seeds multiply the single-run cost.
5. Write the one-page brief and a risk list now; they become the skeleton of the report.

References and further reading

The cost formulas are the standard approximations also used in Modules 5 and 8; every number in this lesson was computed by the code shown. The effective rate of 50 GFLOP/s is an assumption for a laptop CPU, not a measurement of your hardware.

• Raschka, S. (2024). Build a Large Language Model (From Scratch). Manning (the book on which this track is based); companion code at the repository named in the track resources.
• Kaplan, J. et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.
• Hoffmann, J. et al. (2022). Training compute-optimal large language models. arXiv:2203.15556.
• Build a Large Language Model (From Scratch), the whole book (primary book reference)
• Publisher page for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: experiment design and reproducibility: baselines, ablations, seeds and logs.

Related content

Nothing below is required to finish this lesson.

• Before
  LM11-T6: Security, Prompt Injection and Robot Safety

  Recognise injection and unsafe-action risks and keep a safety layer between a model and a robot.

• Lab
  LM12-L1: Capstone Milestone 1: Dataset, Tokenizer and Pipeline

  Prepare a dataset, a tokenizer and a data pipeline with checks.