M1-T5 — Evaluating a Policy

Learning objectives

After this lesson, you should be able to:

• Choose metrics for a policy: success rate, return, safety events and time to goal.
• Compute and read a Wilson interval for a success rate and a t-interval for a mean over seeds.
• Compare two policies on the same starts and say what the paired counts show.
• Work out how many trials are needed to tell a 90% policy from an 80% one.

The intuition

If you tossed a coin 20 times and got 18 heads you would not claim it lands heads 90% of the time; you would say “probably more than half, I am not sure how much more”. A policy trial is a coin toss with an unknown bias. The success rate you measure is an estimate, and with few trials it is a rough one.

Two things can make the estimate wrong. The first is noise: a different set of 20 starts would give a different count. An interval measures that. The second is bias: if the starts are not like the ones the robot will meet, or if the policy has already seen them, no number of trials repairs it. Intervals do not cure bias; held-out scenarios do.

The mathematics

What to measure

The unit of evaluation

One trial is one episode from one start, run to success, a safety event or the step limit. Trials are independent only when the starts are chosen independently of the policy. Starts that are held out must never appear in training data or have been used to choose features or settings.

Interval for a success rate

With k successes in n trials, the simple estimate is p = k / n. The Wilson 95% interval is:

Unlike the textbook p ± z·sqrt(p(1−p)/n), it stays inside [0, 1] and behaves sensibly when p is 0 or 1. Always use it for rates near the ends, which is where robots live.

Interval for a mean over seeds

If training is repeated with s different random seeds and each gives a score x_i, the spread between seeds is the training noise. A 95% interval for the mean uses Student’s t:

sd

sample standard deviation of the s scores (divide by s − 1).

t

critical value; 2.262 for 9 degrees of freedom, larger for fewer seeds.

seed

the number that fixes every random choice: initial weights, data order, simulator noise.

Paired comparison

To compare policies A and B, run both from the same starts. Count the starts where only A succeeded (a), only B succeeded (b), both (c) and neither (d). Only the discordant counts a and b carry information about which is better: if the policies were equally good, a and b would be about equal. Pairing removes the noise from easy and hard starts, because both policies face them.

How many trials?

For two independent groups with success rates p₁ and p₂, 95% confidence (two-sided) and 80% power, a normal-approximation rule gives per group:

A worked example

All numbers below were computed by running RLCore on Scout’s world. The policy is a linear map on pillar-aware features fitted to 30 expert demonstrations (lesson M1-T4); its twin uses unit-direction features only. “Held out” is 100 starts from the hall’s left third (starts(500, 100, 'all')), disjoint from the 30 training starts.

1. Intervals for a rate. 18 of 20 gives Wilson [0.699, 0.972]; 180 of 200 gives [0.851, 0.934]. Both have p = 0.90, but the first interval is 0.273 wide and the second 0.083. The twenty-trial run cannot say whether the policy is 75% or 95% reliable. A perfect result is not exact either: 100 of 100 gives [0.963, 1.000], and the true rate may still be below 0.97.

2. Training scores lie. The direction-feature policy fitted to expert demonstrations from easy starts (far from the pillar line) reaches the goal in 30 of 30 training starts. On the 100 held-out starts it reaches it in 45, and from starts near the pillar line in 0 of 100. The training score says nothing about the starts the hall will actually produce.

3. Paired comparison. Train both feature sets on the same 30 demonstrations and run both from the same 100 held-out starts:

Direction features: 45 of 100, Wilson [0.356, 0.548], mean return 0.77, 52 crashes. Pillar features: 100 of 100, Wilson [0.963, 1.000], mean return 8.45, no crashes. The intervals do not overlap, and the paired table is stronger still: of the 55 starts where the policies differ, the pillar-aware one won all 55 and lost none.

4. Over seeds. Repeat the training of the direction policy with 10 different sets of 30 demonstrations (seeds 1 to 10) and score each on the same 100 starts. The success rates are 0.45, 0.45, 0.45, 0.39, 0.47, 0.43, 0.39, 0.46, 0.44 and 0.41: mean 0.434, standard deviation 0.028. The interval is 0.434 ± 2.262 × 0.028 / √10 = 0.434 ± 0.020 = [0.414, 0.454]. The seed-to-seed spread is small beside the gap to the pillar-aware policy, so the difference is real; the pillar-aware policy scored 1.00 for all ten seeds.

5. How many trials for 90% against 80%? With p₁ = 0.9, p₂ = 0.8: p̄ = 0.85, so n = (1.96 × √(2 × 0.85 × 0.15) + 0.8416 × √(0.09 + 0.16))² / 0.01 = (0.9898 + 0.4208)² / 0.01 ≈ 199 per policy. With 100 trials each, a single 90% estimate has a half-width of about 0.059, so 90 against 80 can easily overlap. To separate 90% from 85% the same formula needs about 686 per policy. Small differences are expensive; this is why “one point better” is rarely an answer.

What the simulation simplifies. Scout’s hall has no noise, no moving people and one obstacle, so the policies are deterministic and the only randomness is the choice of starts and demonstrations. A real robot also varies from run to run with the same start, which widens every interval.

Engineering implications and limitations

Fix the evaluation before training

Choose the held-out scenarios, the metrics and the stopping rule first, and write them down. Looking at results first and choosing the metric second is a quiet form of tuning on the test set.

Cover the hard cases on purpose

A random sample of starts mostly contains easy ones. Add stratified sets: near the pillar, narrow gaps, the lowest battery. Report each stratum, because an average over strata hides the one where the policy fails.

Report counts, not just rates

Write “18 / 20 (Wilson 95%: 0.70 to 0.97)” rather than “90%”. State n, the number of seeds, and whether seeds refer to training, to the environment or to both.

Separate safety events

Count collisions and near misses on their own line. A policy with a higher return and one more crash per hundred episodes is not obviously better.

Beware of the best seed

Reporting the best of ten training runs measures luck. Report all of them, or the mean with its interval, as argued by Henderson et al. and Agarwal et al.

Common misconceptions

Engineering takeaways

1. Measure success rate, return, safety events and time on held-out scenarios, and report them separately.
2. A Wilson interval gives the honest range for a success rate; 18/20 is [0.70, 0.97] and 180/200 is [0.85, 0.93].
3. Over seeds, report the mean with a t-interval; with 10 seeds the critical value is 2.262.
4. Compare policies on the same starts and read the discordant counts.
5. Telling 90% from 80% needs about 200 trials per policy, and training-set success tells you nothing about the rest.

References and further reading

Primary sources on reporting reinforcement learning results and on the interval for a proportion.

• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup and D. Meger, “Deep Reinforcement Learning That Matters”, AAAI, 2018.
• R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville and M. G. Bellemare, “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, NeurIPS, 2021.
• E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference”, Journal of the American Statistical Association 22(158), 1927.
• L. D. Brown, T. T. Cai and A. DasGupta, “Interval Estimation for a Binomial Proportion”, Statistical Science 16(2), 2001.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: when the numbers are bad, a field guide to why learned policies fail and the first thing to check for each.

Related content

Nothing below is required to finish this lesson.

• Before
  M1-T4: Training Versus Inference

  Separate what is learned from what is fixed, and what runs on the robot from what runs on a workstation.

• Lab
  M1-L3: Score a Policy Fairly

  Two policies both "work": compute success rates with intervals on held-out scenarios and decide which one is really better.