LM11-T5 — Application Evaluation, Latency, Cost and Reliability

Learning objectives

After this lesson, you should be able to:

• Define task success, component metrics and a labelled evaluation set for an application.
• Estimate latency (time to first token, total, percentiles) and cost per request and per month.
• Compute the reliability of chained steps and the effect of retries.
• Report pass rates with Wilson confidence intervals and design regression and monitoring practices.

The idea: measure what the user gets, then find the stage that limits it

Start from the user’s goal, not the model’s score. For a retrieval assistant the goal is “the operator gets a correct, sourced answer quickly, or is told that the system does not know”. Everything else is a means. Define success in a way that can be checked: for each test question, a reference answer or a rubric, and a rule that says pass or fail.

Three layers of measurement

The evaluation set

Use real or realistic requests, include the difficult and the hostile (unanswerable questions, ambiguous requests, injection attempts, malformed inputs), keep it separate from anything used to tune prompts, and keep it versioned. 100 well-chosen cases usually reveal most problems; 1,000 are needed to detect differences of a few percentage points (Module 9). When the answers are free text, use a rubric and, where you use another model as a judge, check the judge against human labels first.

Variability

Sampling, model updates, retrieval ties and external services make results vary. Run important comparisons on the same cases, report intervals, and run enough repeats to see the spread. A prompt change that raises the pass rate from 80% to 83% on 100 cases is not evidence (the intervals overlap widely).

Latency is a distribution

Users feel the slow tail. Report median, 95th and 99th percentiles, not only the mean. In a list of ten latencies 1.1, 1.2, 1.2, 1.3, 1.3, 1.4, 1.5, 1.6, 2.0 and 7.5 seconds the mean is 2.01 s, the median 1.3 s and the 95th percentile 7.5 s; each tells a different story, and the outlier is what produces complaints. Causes of the tail: long generations, retries, cold caches, queueing, slow tools.

Cost

Providers and self-hosted systems both charge, in dollars or in GPU time, for prompt tokens (prefill) and generated tokens (decode). Retries multiply it, long prompts inflate it, and multi-step agents compound it. Budget with the actual token counts from logs, not with guesses.

Evaluate before you scale

The mathematics: budgets, chains and intervals

Cost

Illustrative prices (made up for the arithmetic, not any provider’s): 0.50 dollars per million input tokens, 1.50 per million output tokens. A request with 1,500 prompt tokens and 200 reply tokens costs (750 + 300)/10⁶ = 0.00105 dollars; with one retry 0.0021. For 100,000 requests: 105 dollars; a retry rate of 10% adds about 10.5 dollars. Halving the prompt by trimming examples saves 37.5 dollars here, more than most model changes.

Latency

With r_prefill = 5,000 tokens/s and r_decode = 60 tokens/s: time to first token 1,500/5,000 = 0.3 s, total 0.3 + 200/60 = 3.63 s. The reply length, not the prompt length, dominates here; limiting the answer to 100 tokens saves 1.67 s. These rates come from measurements of your own system (Lesson LM9-T5 explains why decoding is memory-bound).

Chained steps

If a task needs n steps and step i succeeds with probability p_i (independently), the whole succeeds with Π p_i. For five steps of equal reliability: 0.99⁵ = 0.951, 0.95⁵ = 0.774, 0.90⁵ = 0.590. Agents with many steps need very reliable steps, or checkpoints where failures are detected early and handled, or fewer steps.

Confidence intervals for a pass rate

With k passes in n trials, the Wilson interval is better behaved than the normal approximation at small n or rates near 0 or 1:

For 18 of 20: (0.699, 0.972). For 90 of 100: (0.826, 0.945). For 900 of 1,000: (0.880, 0.917). The same 90% looks very different at different n: with 20 cases you know only that the true rate is between 70% and 97%.

A worked example: the application calculator

The code reproduces every number of this lesson: cost per request and per 100,000 requests, latency from prefill and decode rates, chained success, Wilson intervals and latency percentiles. Replace the constants with measurements from your system.

import math

def request_cost(prompt_tokens, reply_tokens, price_in=0.5, price_out=1.5, retries=0):
    """Dollars per request; prices are per million tokens (illustrative)."""
    calls = 1 + retries
    return calls * (prompt_tokens * price_in + reply_tokens * price_out) / 1e6

print(round(request_cost(1500, 200), 6), round(request_cost(1500, 200, retries=1) * 1e6))   # 0.00105 2100
print(round(request_cost(1500, 200) * 100_000, 2))      # 105.0  dollars per 100,000 requests

def latency(prompt_tokens, reply_tokens, prefill_tps=5000, decode_tps=60):
    ttft = prompt_tokens / prefill_tps
    return ttft, ttft + reply_tokens / decode_tps

ttft, total = latency(1500, 200)
print(round(ttft, 2), round(total, 2))                 # 0.3 3.63

def chain_success(p, steps):
    return p ** steps

print([round(chain_success(p, 5), 3) for p in (0.99, 0.95, 0.9)])   # [0.951, 0.774, 0.59]

def wilson(successes, n, z=1.96):
    p = successes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return round(centre - half, 3), round(centre + half, 3)

print(wilson(18, 20), wilson(90, 100), wilson(900, 1000))   # (0.699, 0.972) (0.826, 0.945) (0.88, 0.917)

def percentile(values, q):
    s = sorted(values)
    return s[min(len(s) - 1, int(math.ceil(q * len(s))) - 1)]

lat = [1.1, 1.2, 1.2, 1.3, 1.3, 1.4, 1.5, 1.6, 2.0, 7.5]
print(round(sum(lat) / len(lat), 2), percentile(lat, 0.5), percentile(lat, 0.9), percentile(lat, 0.95))   # 2.01 1.3 2.0 7.5

Monitoring after launch

1. Log structured records,Prompt version, model version, retrieved ids, raw reply, validation result, retries, latency, tokens and cost. Mind privacy and retention rules.
2. Track the numbers that predict trouble,Validation failure rate, retry rate, refusal rate, empty retrievals, p95 latency, cost per request.
3. Sample and review,Have humans read a random sample and all flagged cases each week; add failures to the regression set.
4. Alert on change,A jump in retries or in empty retrievals often signals an upstream change (a model update, a broken index).
5. Keep a rollback,Version prompts, models and indexes so a bad change can be reverted in minutes.

Engineering implications and limitations

Write the success rule before building. “Correct answer with a valid citation, or an explicit don't-know, within 5 seconds at the 95th percentile, under 0.5 cents” is a testable requirement; “works well” is not.

Evaluate stage by stage and end to end. Use the decomposition accuracy = r·a + (1 − r)·b (Lesson LM11-T3) to decide where to work.

Report intervals, not points. Use Wilson intervals or the paired methods of Module 9, and do not ship on a difference smaller than the noise.

Keep a regression set that only grows. Every production failure becomes a test case; every change must pass the whole set.

Budget worst cases. Longest allowed prompt, longest reply, maximum retries, maximum steps. That product is the real upper bound on cost and latency.

Separate quality from speed trade-offs. When you quantise, shorten prompts or switch models to save money, measure quality and cost together (Lesson LM10-T6).

For robots. Add a hard deadline to the budget and a safe default action for the case when the language-model path misses it. Measure the rate at which the fallback is used; a rising rate is an early warning.

Common misconceptions

Engineering takeaways

1. Define success from the user’s goal and measure components, the end-to-end task and operation (latency, cost) separately.
2. Cost per request = (1 + retries)·(n_in·price_in + n_out·price_out)/10⁶; latency ≈ retrieval + n_in/r_prefill + n_out/r_decode.
3. Chained steps multiply: 0.95⁵ = 0.774. Reduce steps, raise step reliability or add checkpoints.
4. Report Wilson intervals; 18/20 is (0.70, 0.97) while 900/1000 is (0.88, 0.92). Report latency percentiles.
5. Keep versioned evaluation and regression sets, log structured records and monitor retries, empty retrievals, p95 latency and cost.

References and further reading

Module 9 of this track covers confidence intervals, paired tests and evaluation harnesses in detail. The prices, token rates and latencies in this lesson are illustrative figures chosen for the arithmetic; they are not quotations from any provider or measurements of a real system.

• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158).
• Liang, P. et al. (2022). Holistic evaluation of language models. arXiv:2211.09110.
• Zheng, L. et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: security, prompt injection and robot safety: what can go wrong when a model reads untrusted text and can act.

Related content

Nothing below is required to finish this lesson.

• Before
  LM11-T4: Tool Calling and Schema Validation

  Let a model request tools through a validated schema and keep the application in control.

• Lab
  LM11-L3: Build a Retrieval Question-Answering Prototype

  Build retrieval and prompt assembly for a small document set, compare answers with and without context and test an injection case.