8.2 — Task Success Rate and Repeatability

Learning objectives

After this lesson, you should be able to:

• Explain why a success criterion must be defined in advance.
• Show how repeated evaluations of one policy differ.
• Compute the trials needed for a margin of error.
• Estimate the smallest detectable difference between two policies.

The one-sentence idea

A rate needs a definition and a trial count.

Small differences need many trials.

First, define “success”

Success rate sounds simple until you ask what counts. Is the object “in the bowl” when it touches the rim? Must it stay for five seconds? How close to the target counts? OpenVLA's headline comparison, 16.5 percentage points of absolute task success over RT-2-X across 29 tasks and several embodiments, only means something because those tasks had defined success criteria.^[1] The same behaviour can produce quite different numbers under different definitions:

import numpy as np
rng = np.random.default_rng(0)

final_error = np.abs(rng.normal(0.05, 0.04, 500))            # distance from the goal at the end of 500 trials (metres)
for tolerance in (0.01, 0.03, 0.05, 0.10):
    print(f"success means within {tolerance*100:4.0f} cm -> success rate {np.mean(final_error < tolerance):.2f}")

success means within    1 cm -> success rate 0.13
success means within    3 cm -> success rate 0.30
success means within    5 cm -> success rate 0.51
success means within   10 cm -> success rate 0.89

Toy model. Invented final errors. The point: a rate is meaningless without its criterion.

Write the criterion down before the trials, with the measurement method, and report it with the result.

Repeatability: the same policy, evaluated again

Run an evaluation of 30 trials, then run it again with the same policy. You will not get the same number, because the world and the policy both contain chance. How much it varies is the evaluation's repeatability:

import numpy as np
rng = np.random.default_rng(0)

true_rate = 0.70
trials = 30
rates = [rng.binomial(trials, true_rate) / trials for _ in range(10)]
print("ten evaluations of the SAME policy:", [round(r, 2) for r in rates])
print("lowest and highest:", round(min(rates), 2), round(max(rates), 2))

ten evaluations of the SAME policy: [0.67, 0.77, 0.83, 0.87, 0.63, 0.6, 0.67, 0.67, 0.7, 0.57]
lowest and highest: 0.57 0.87

A policy with a true rate of 0.70 produces measured rates spanning more than 20 points just because of luck. If you change something and the number moves by 10 points, that may be nothing at all (lessons 3.4 and 7.6).

How many trials do I need?

The number of trials sets the precision. For a rate near p, the 95% margin of error is about 1.96·√(p(1−p)/n). Solving for n gives the trials needed for a chosen margin of error:

import numpy as np

p = 0.5                                      # the worst case: the margin is widest at 50%
print("margin of error (95%) | trials needed")
for margin in (0.20, 0.10, 0.05, 0.02):
    n = int(np.ceil(1.96**2 * p * (1 - p) / margin**2))
    print(f"        +/- {margin*100:4.0f} points | {n:>5}")

margin of error (95%) | trials needed
        +/-   20 points |    25
        +/-   10 points |    97
        +/-    5 points |   385
        +/-    2 points |  2401

±10 points needs about 100 trials; ±5 points about 400; ±2 points about 2,400. A real robot lab can rarely afford the last. This is also why simulation benchmarks (lesson 8.1) and SIMPLER-style simulated evaluations, which exist because real-world evaluation is not scalable,^[2] are valuable for cheap, repeatable trials.

How big a difference can I detect?

Comparing two policies is harder than measuring one. With n independent trials each and rates near 0.5, the smallest difference that clears the noise is roughly 2.77·√(p(1−p)/n):

import numpy as np

p = 0.5
for n in (20, 50, 100, 400):
    smallest = 1.96 * np.sqrt(2) * np.sqrt(p * (1 - p) / n)
    print(f"{n:>4} trials each -> differences below about {smallest*100:4.1f} points can be noise")

  20 trials each -> differences below about 31.0 points can be noise
  50 trials each -> differences below about 19.6 points can be noise
 100 trials each -> differences below about 13.9 points can be noise
 400 trials each -> differences below about  6.9 points can be noise

With 20 trials each, a gap of 30 points could be luck. Pairing trials on the same scenes (lesson 7.7) helps, but the basic message stands: small improvements need many trials, and honest papers say how many were run.

Common misconceptions

Engineering takeaways

1. Define success first.
2. Repeat evaluations vary.
3. ±5 points needs about 400 trials.
4. Report n.

Say what counted, and how many times you tried.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: testing generalisation to new objects, positions and instructions.

Back to the VLA roadmap