8.1 — LIBERO and Other Simulation Benchmarks

Learning objectives

After this lesson, you should be able to:

• Describe the structure of LIBERO’s four suites.
• Compute micro and macro averages and explain the difference.
• Show how selecting the best variant inflates a score.
• State what benchmarks cannot tell you.

The one-sentence idea

A benchmark compares methods on common ground, if used carefully.

Look at the weighting and the selection process behind any score.

What a benchmark is for

A benchmark is a fixed set of tasks, scenes and rules that everyone runs, so results can be compared. Real-robot evaluation is not scalable and has reproducibility problems, which is the starting point of SIMPLER, a suite of simulated environments that mirror real evaluation setups (RT-1 and Bridge-V2) and whose paired sim-and-real evaluations show strong correlation between the two.^[1] Another widely used simulation benchmark is LIBERO, built for studying lifelong learning of manipulation: 130 tasks in four procedurally generated suites, LIBERO-Spatial, LIBERO-Object and LIBERO-Goal with 10 tasks each (targeting spatial relationships, objects and task goals) and LIBERO-100 with 100 tasks, with human-teleoperated demonstrations for all of them.^[2][2]

Benchmarks make progress measurable, but they also make it easy to fool yourself. This lesson covers what a suite's structure tells you, how averaging changes the story, and how repeated peeking inflates a score.

Read the structure first

Each suite of LIBERO isolates a kind of knowledge: where things are, which object, what goal. A policy that scores well on one suite and badly on another is telling you something. Reading the structure from the description, here is how the 130 tasks divide:

suites = {"LIBERO-Spatial": 10, "LIBERO-Object": 10, "LIBERO-Goal": 10, "LIBERO-100": 100}
total = sum(suites.values())
print("total tasks:", total)
for name, n in suites.items():
    print(f"{name:<15} {n:>3} tasks  ({100 * n / total:4.1f}% of all tasks)")

total tasks: 130
LIBERO-Spatial   10 tasks  ( 7.7% of all tasks)
LIBERO-Object    10 tasks  ( 7.7% of all tasks)
LIBERO-Goal      10 tasks  ( 7.7% of all tasks)
LIBERO-100      100 tasks  (76.9% of all tasks)

Notice that LIBERO-100 is 77% of the tasks. A single number averaged over all 130 tasks would mostly describe LIBERO-100, so the choice of how to average matters.

Micro versus macro averages

Averaging over tasks (micro) lets the biggest suite dominate. Averaging over suites (macro) gives each suite equal say. Neither is wrong, but they answer different questions, and a paper should say which it used. With invented success rates:

results = {"LIBERO-Spatial": (10, 0.90), "LIBERO-Object": (10, 0.95), "LIBERO-Goal": (10, 0.85), "LIBERO-100": (100, 0.55)}   # (tasks, mean success), invented

micro = sum(n * s for n, s in results.values()) / sum(n for n, s in results.values())
macro = sum(s for n, s in results.values()) / len(results)
print("average over all tasks (micro):", round(micro, 3))
print("average over suites   (macro):", round(macro, 3))

average over all tasks (micro): 0.631
average over suites   (macro): 0.812

Toy model. The success rates are invented to show the arithmetic; they are not results for any real policy.

A policy that is strong on three small suites and weak on the big one looks good under macro and mediocre under micro. When you read a headline number, look for the weighting.

Overfitting to the benchmark

If you try many variants of a method and report the best test score, the score is biased upward by the selection. The effect is easy to see when every variant is exactly as good as the others:

import numpy as np
rng = np.random.default_rng(0)

true_rate = 0.60                  # every variant is truly 60% successful
trials = 50                       # trials used to measure each variant
for variants in (1, 5, 20, 100):
    best = [max(rng.binomial(trials, true_rate, variants)) / trials for _ in range(2000)]
    print(f"best of {variants:>3} variants: reported success {np.mean(best):.3f}   (true rate {true_rate})")

best of   1 variants: reported success 0.600   (true rate 0.6)
best of   5 variants: reported success 0.680   (true rate 0.6)
best of  20 variants: reported success 0.726   (true rate 0.6)
best of 100 variants: reported success 0.768   (true rate 0.6)

With 100 equally good variants the reported number is well above the truth. The remedy is a separate set of tasks or seeds that you look at only once, at the end. Many benchmarks provide fixed splits for this; use them, and never tune on the final test.

What a benchmark cannot tell you

A benchmark measures the tasks it contains, in the simulator it uses. It says little about your robot, your objects or your lighting. SIMPLER's own motivation is that a simulation of a real setup needs checking against reality, which it does with paired evaluations.^[1] Use benchmarks to compare methods on a common footing, and your own task-specific tests (lessons 7.6 and 7.7) to decide whether a method is good enough for you.

Common misconceptions

Engineering takeaways

1. LIBERO: 130 tasks, four suites.
2. Micro vs macro averages differ.
3. Do not tune on the final test.
4. Benchmarks are not your robot.

Ask how the number was made.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: measuring task success and repeatability.

Back to the VLA roadmap