7.7 — Simulation-Based Manipulation Experiments

Learning objectives

After this lesson, you should be able to:

• Generate repeatable scenes from seeds.
• Compare two policies on shared scenes.
• Run a one-factor sweep.
• Check whether simulation ranks policies like the real robot.

The one-sentence idea

Fix the scenes, change one thing, compare in pairs.

Repeatable and paired.

Why experiment in simulation

Real-robot trials are slow, costly and hard to repeat. Simulation is fast, free and exactly repeatable, which makes it the right place for controlled experiments, with one caveat: it is not the real world (the reality gap, lesson 1.1^[1]). Simulators such as MuJoCo are physics engines for control: MuJoCo handles multi-joint dynamics in generalised coordinates and uses velocity-stepping algorithms for contact, with models written in an XML format or through a C++ API.^[2] Benchmarks package scenes and tasks on top of an engine. LIBERO, for example, has 130 tasks in four suites (LIBERO-Spatial, -Object and -Goal with 10 tasks each, and LIBERO-100) with human teleoperated demonstrations.^[3] And SIMPLER builds simulated versions of real evaluation setups (RT-1 and Bridge-V2) after finding that real-world evaluation is not scalable and has reproducibility problems.^[4]

This lesson is about the experimental method, which applies to any simulator. The cells use a tiny simulated reaching task so you can run them here; lesson 8.1 covers benchmarks in more detail.

Repeatability: seeds and scenes

A trial is a scene plus a policy. If the scene comes from a seed, anyone can regenerate the exact same scenes. That makes comparisons fair (same scenes for both policies) and failures reproducible (rerun scene 17):

import numpy as np

def make_scene(seed):
    rng = np.random.default_rng(seed)
    return {"hand": rng.uniform(-1, 1, 2).round(3).tolist(), "goal": rng.uniform(-1, 1, 2).round(3).tolist(), "light": round(float(rng.uniform(0.5, 1.5)), 2)}

print("scene 17:", make_scene(17))
print("scene 17 again:", make_scene(17))
print("identical:", make_scene(17) == make_scene(17))
print("scene 18:", make_scene(18))

scene 17: {'hand': [0.69, -0.678], 'goal': [0.115, -0.264], 'light': 0.71}
scene 17 again: {'hand': [0.69, -0.678], 'goal': [0.115, -0.264], 'light': 0.71}
identical: True
scene 18: {'hand': [-0.201, 0.435], 'goal': [-0.438, -0.835], 'light': 1.47}

Log the seed with every result. “Fails on seed 17” is a debugging handle; “fails sometimes” is not.

Fair comparison: the same scenes for both policies

Run policy A and policy B on the same set of seeds. Then count, per scene, who succeeded. Only the scenes where they disagree tell you anything about which is better:

import numpy as np

def run(policy_gain, seed, noise=0.05):
    rng = np.random.default_rng(seed)
    hand, goal = rng.uniform(-1, 1, 2), rng.uniform(-1, 1, 2)
    for _ in range(10):
        hand = hand + policy_gain * (goal - hand) + rng.normal(0, noise, 2)
    return bool(np.linalg.norm(hand - goal) < 0.15)

seeds = range(200)
a = np.array([run(0.30, s) for s in seeds])               # policy A
b = np.array([run(0.45, s) for s in seeds])               # policy B reaches faster

both, neither = int(np.sum(a & b)), int(np.sum(~a & ~b))
only_a, only_b = int(np.sum(a & ~b)), int(np.sum(~a & b))
print("A success rate:", a.mean(), "| B success rate:", b.mean())
print("both succeed:", both, "| neither:", neither, "| only A:", only_a, "| only B:", only_b)

A success rate: 0.875 | B success rate: 0.94
both succeed: 174 | neither: 11 | only A: 1 | only B: 14

Toy model. A one-line controller with two settings. The structure of the comparison, not the policies, is the point.

The scenes in which only one policy succeeded carry the evidence. If “only B” is far larger than “only A”, B is better on these scenes; if they are close, the difference between the success rates is probably noise. This is a sign test on the disagreements, the paired version of the bootstrap in lesson 7.6.

Randomising on purpose

Varying the scene deliberately, in lighting, positions and object properties, is called randomisation. The domain randomisation paper trained a detector only on simulated images with randomised, non-realistic textures and localised real objects to about 1.5 cm; the argument is that with enough variability in the simulator the real world looks like just another variation.^[5] In evaluation, the same idea lets you measure how success changes along one axis (lesson 5.5):

import numpy as np

def run(seed, noise):
    rng = np.random.default_rng(seed)
    hand, goal = rng.uniform(-1, 1, 2), rng.uniform(-1, 1, 2)
    for _ in range(10):
        hand = hand + 0.3 * (goal - hand) + rng.normal(0, noise, 2)
    return bool(np.linalg.norm(hand - goal) < 0.15)

for noise in (0.02, 0.05, 0.10, 0.20):                      # one factor: how noisy the actuators are
    rate = np.mean([run(s, noise) for s in range(300)])
    print(f"actuator noise {noise:.2f}: success {rate:.2f}")

actuator noise 0.02: success 1.00
actuator noise 0.05: success 0.87
actuator noise 0.10: success 0.44
actuator noise 0.20: success 0.16

One factor at a time gives a curve, and a curve shows where the policy starts to break.

How far to trust the simulator

The question is always whether the simulator ranks policies the way reality does. SIMPLER reports a strong correlation between simulated and real performance in its paired evaluations.^[4] You can compute the same kind of check yourself: rank your policies in simulation and on the real robot, and measure how well the orderings agree with a rank correlation (1 means identical order, −1 reversed):

import numpy as np

sim  = np.array([0.80, 0.65, 0.50, 0.40, 0.30])             # invented success rates of five policies in simulation
real = np.array([0.55, 0.60, 0.35, 0.20, 0.25])             # invented success rates of the same policies on the robot

def ranks(x):
    return np.argsort(np.argsort(x))

def spearman(a, b):
    ra, rb = ranks(a), ranks(b)
    return float(np.corrcoef(ra, rb)[0, 1])

print("rank correlation, simulation vs real:", round(spearman(sim, real), 2))
print("the simulator overestimates every policy:", bool(np.all(sim > real)))

rank correlation, simulation vs real: 0.8
the simulator overestimates every policy: True

Toy model. The success rates are invented to show the calculation. They are not SIMPLER’s numbers.

Here the simulator is optimistic about every policy yet mostly orders them as the robot does (correlation 0.8): the middle policy agrees, while the top two and the bottom two each swap places. That is the kind of agreement you need for simulation to guide choices, though with five policies it is weak evidence. If the ranking does not hold, do not use simulation to choose; use it only to catch gross failures.

Common misconceptions

Engineering takeaways

1. Seeds make scenes repeatable.
2. Paired comparison counts disagreements.
3. Sweep one factor.
4. Validate the simulator.

Same scenes, one change.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the implementation phase. Next: evaluating and generalising, starting with benchmarks.

Back to the VLA roadmap