LM9-T5 — Inference Latency and Memory

Learning objectives

After this lesson, you should be able to:

• Distinguish prefill from decode and the metrics of each.
• Estimate time to first token from FLOPs and tokens per second from memory bandwidth.
• Explain the effect of batching, quantisation and the KV cache on throughput and memory.
• Measure latency properly: warm-up, repetition, percentiles.

The idea: two phases, two bottlenecks

To answer a request the system first reads the whole prompt, then writes the answer one token at a time.

Prefill

All prompt tokens are processed in a single parallel pass (they are known in advance, so the causal mask allows it). The work is about 2·P FLOPs per token, spread over many tokens at once: the hardware is busy with arithmetic. The time until the first output token is the time to first token (TTFT); it grows roughly in proportion to the prompt length.

Decode

Each new token needs a pass through the whole model, but only for one position. The arithmetic is tiny; the cost is reading every weight from memory to do it. So the speed of decoding, in tokens per second, is set by how fast the hardware can stream the weights: memory bandwidth divided by the model size in bytes. Smaller weights (quantisation) directly speed it up.

Batching

If several requests are decoded together, the weights are read once per step for all of them. Throughput (total tokens per second) rises almost in proportion to the batch size until the arithmetic becomes the limit; latency per request rises only slightly. The price is memory: each request has its own KV cache.

The KV cache

At each decode step the model needs the keys and values of all previous tokens. Recomputing them would repeat the prefill at every step, so they are stored (Module 10). The cache grows with the sequence and with the number of concurrent requests. Reading it is also memory traffic in every step, so long contexts slow down decoding.

What to measure

• TTFT (for interactivity) and time per output token (for streaming speed).
• Throughput: tokens per second across all requests, at a stated batch size.
• Memory: weights plus KV cache at the target concurrency and context length.
• Tail latency: the 95th and 99th percentile, not only the mean.

The mathematics: roofline estimates

Let the model have P parameters stored at b bytes each, run on hardware with peak arithmetic rate F (FLOP/s) at utilisation u, and memory bandwidth W (bytes/s).

n_prompt

number of prompt tokens.

F · u

achieved arithmetic rate; u is typically 0.3 to 0.6 for large matrix products.

W

memory bandwidth; ~10¹² bytes/s for a high-end accelerator.

t_step

time of one decode step for a single sequence; the weights are read once.

Example (7B model). P = 7·10⁹; F = 10¹⁴ FLOP/s at u = 0.4; W = 10¹² B/s. A 1,000-token prompt: 2 · 7·10⁹ · 1,000 / (4·10¹³) = 0.35 s. Decoding in 16 bits (b = 2): t_step = 14 GB / 1 TB/s = 14 ms, 71 tokens/s. In 8 bits: 7 ms, 143 tokens/s. In 4 bits: 3.5 ms, 286 tokens/s.

Batching. With batch B and per-request KV cache of K bytes, a step reads the weights once and each request’s cache: bytes per step = P · b + B · K, and arithmetic 2 · P · B. The step is memory-bound while

For the numbers above (16-bit): B* = 4·10¹³ · 2 / (2 · 10¹²) = 40. Below 40 concurrent sequences, doubling the batch nearly doubles throughput. With B = 8 and a 1,000-token cache of 0.5 MB per token (500 MB per request): bytes per step = 14 GB + 4 GB = 18 GB, 18 ms per step for 8 tokens, 444 tokens/s in total (56 per request) against 71 for a single request.

Memory. Serving memory = P · b + B · n · 2 · L · d · b_KV (Lesson LM8-T3). For the 7B model: 14 GB of weights in 16 bits; the cache of 0.52 MB per token at 16 bits means 8 requests of 4,096 tokens need 17 GB, more than the weights.

Tail latency. Real services see queueing and variable lengths. If the service rate is μ requests per second and the arrival rate λ, utilisation ρ = λ/μ, a simple M/M/1 model gives a mean time in the system of 1/(μ − λ): at ρ = 0.9 it is ten times the service time, at ρ = 0.5 twice. Plan capacity for headroom, not for 100% utilisation.

A worked example: latency calculators

The code reproduces the estimates above and a batched-throughput table, then shows a measurement helper that avoids the usual mistakes (warm-up, repeated runs, percentiles).

import statistics
import time

P, F, U, W = 7e9, 1e14, 0.4, 1e12            # parameters, FLOP/s, utilisation, bytes/s

def ttft(nprompt):
    return 2 * P * nprompt / (F * U)

def tokens_per_second(bytes_per_weight, batch=1, kv_bytes_per_request=0.0):
    step = (P * bytes_per_weight + batch * kv_bytes_per_request) / W        # memory-bound step
    compute = 2 * P * batch / (F * U)                                       # arithmetic for the batch
    return batch / max(step, compute)

print(round(ttft(1000), 2))                                                  # 0.35
for b in (2, 1, 0.5):
    print(b, round(tokens_per_second(b)))                                       # 2 71 / 1 143 / 0.5 286
print(round(tokens_per_second(2, batch=8, kv_bytes_per_request=0.5e6 * 1000)))               # 444
print(round(F * U * 2 / (2 * W)))                                                           # 40  batch where decode turns compute-bound
for batch in (1, 8, 32, 64):
    print(batch, round(tokens_per_second(2, batch)))                         # 71 571 2286 2857 (compute limit at large batch)

def measure(fn, warmup=2, runs=20):
    for _ in range(warmup):
        fn()                                                                 # warm-up: caches, compilation, allocator
    times = []
    for _ in range(runs):
        t0 = time.perf_counter()
        fn()
        times.append(time.perf_counter() - t0)
    times.sort()
    return statistics.median(times), times[int(0.95 * len(times)) - 1]

median, p95 = measure(lambda: sum(i * i for i in range(20000)))
print(median > 0 and p95 >= median)                                          # True

The batch table shows the regime change: throughput grows almost linearly with batch size until about 40 sequences and then flattens at the arithmetic limit (2,857 tokens/s for these numbers), while the cache grows linearly with batch size.

What to change when

Engineering implications and limitations

Measure the thing the user feels. TTFT and time per output token at the real prompt and output lengths, not a synthetic short prompt.

Warm up and repeat. The first calls include one-off costs (allocator, kernel compilation, caches). Discard them, then take many runs and report median and high percentiles.

Synchronise on accelerators. GPU calls are asynchronous; call the framework’s synchronise function before reading the clock, or you time only the launch.

State the batch size and the hardware. “71 tokens/s” means nothing without them.

Separate model quality from serving speed when quantising. Measure the accuracy loss and the speed gain together (Lesson LM8-T4); a 4-bit model that is 4× faster but 5 points worse may or may not be a good trade.

Limit output length. Decode time grows linearly with the output, so a maximum length caps tail latency and cost.

Estimate before buying hardware. The roofline numbers tell you whether you need more bandwidth (decode-heavy) or more compute (prefill-heavy, large batches).

For robots. A control loop has a deadline. If the model must return a command in 200 ms, budget TTFT plus the tokens of a structured answer (a dozen tokens) and test on the actual on-board hardware, at its thermal limits; fall back to a rule-based safe action if the deadline is missed.

Common misconceptions

Engineering takeaways

1. Prefill is compute-bound: TTFT ≈ 2·P·n/(F·u) (0.35 s for 1,000 tokens on a 7B model). Decode is memory-bound: tokens/s ≈ W/(P·b) (71, 143, 286 at 16, 8, 4 bits).
2. Batching raises throughput nearly linearly until B* = F·u·b/(2·W) (40 in the example), at the price of KV-cache memory.
3. KV cache = 2·L·d·n·B·bytes; it can exceed the weights and slows long-context decoding.
4. Measure TTFT, time per token, throughput and percentiles, with warm-up, repetitions and synchronisation, at stated batch size and hardware.
5. Choose remedies by bottleneck: quantisation for decode, shorter prompts or prefix caching for prefill, continuous batching for throughput.

References and further reading

See Chapter 4 of the book (parameters and memory of GPT models) and Lesson LM8-T3 for the memory formulas. The hardware figures used here are round illustrative numbers, not the specification of any particular device.

• Williams, S., Waterman, A. and Patterson, D. (2009). Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM 52(4).
• Build a Large Language Model (From Scratch), Chapter 7: evaluating instruction-following models (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: responsible evaluation: how to report what you found, including what you did not test.

Related content

Nothing below is required to finish this lesson.

• Before
  LM9-T4: Hallucination and Prompt Sensitivity

  Define unsupported claims, test paraphrases and measure how answers depend on wording.

• Lab
  LM9-L3: Evaluate a Tiny GPT End to End

  Compare checkpoints and decoding settings on a held-out set, with seeds, confidence intervals and a latency and memory measurement.