M10-T1 — The Reality Gap

Learning objectives

After this lesson, you should be able to:

• Name four sources of the reality gap (dynamics, sensing, latency, contact) and say how each shows up in logged data and in behaviour.
• Decompose a policy’s failure in a changed world by changing one factor at a time, report each success rate with its uncertainty, and explain why the factors interact.
• Explain why a policy trained in one simulator can exploit it, using the closest-approach margin as the example.
• State what the toy world includes (gain, drag, delay, observation noise, process noise) and what it leaves out (contact, friction, wear, anything not parameterised).

The intuition

Scout’s policy in this module has three numbers, θ = {clear, kp, ff}. clear is how far beyond the pillar’s contact circle (0.95 m from the centre) it swings, kp is velocity feedback and ff a feedforward term. "Training in simulation" means choosing θ from a small grid by the score success − 0.01 × steps, averaged over episodes in the simulator (RLCore.trainTheta). Trained in the nominal world, with gain 3, drag 3, no delay and no noise, the grid’s winner is clear = 0.3, kp = 0 for every seed 1 to 5: the tightest path that is still perfect in the simulator. It reaches the goal in all 200 nominal episodes.

The figure shows the same θ, from one start, in three worlds. In the nominal world (blue) the swerve starts near x = 4.1 and Scout passes the pillar at 1.03 m from its centre, which is 0.08 m outside contact. Adding only the floor’s lower gain and higher drag (green) makes Scout slower, 46 steps instead of 31, but it still gets there. Adding only a one-step delay between command and effect (orange) starts the swerve one step, 0.25 s, late: Scout has moved about 0.2 m further toward the pillar by then, more than the 0.08 m margin, and it crashes at step 11. A change that costs time in one place costs the whole mission in another.

Three ideas carry the lesson. A simulator is a set of assumptions, and a policy trained in it can come to depend on the assumptions in ways nobody wrote down. The gap has several independent parts, which differ in how loudly they announce themselves: some make the robot slow, one makes it crash. And because the parts interact, "which factor matters most" depends on which direction you ask the question.

The mathematics

Where the gap comes from

Write the true system as x_t+1 = f_real(x_t, a_t−d) with observations o_t = h_real(x_t) + noise, and the simulator as the same two functions with different parameters or a different structure. The policy was optimised for (f_sim, h_sim) and is run on (f_real, h_real). The gap is every way in which these differ, and it is useful to sort the ways.

The table is deliberately short. Other differences that matter on real robots include rendering and lighting for camera-based policies (the target of the original domain randomization work, lesson M10-T2), terrain, payloads, and unmodelled wiring or software behaviour. I put only the first three in the toy because they can be parameterised by a handful of numbers; the fourth I name so that you remember the toy is silent on it.

The simulator as a family

The toy world has five parameters, w = (gain, drag, delay, obsNoise, noise). With the module-7 dynamics, the velocity on each axis moves by

gain

Acceleration per unit command in m/s² (nominal 3, floor 2.2).

drag

Velocity decay in 1/s (nominal 3, floor 3.4). With a constant command a, the speed settles at gain a / drag: 1.00 m/s nominal, 0.647 m/s on the floor.

delay

Whole steps of 0.25 s between the command being issued and acting (nominal 0, floor 1).

obsNoise

Standard deviation of Gaussian noise on the position and velocity readings (nominal 0, floor 0.05).

noise

Standard deviation of process noise on the velocity update (nominal 0, floor 0.03).

ξ, η

Independent standard normal draws.

The nominal world is the point (3, 3, 0, 0, 0); the floor is the point (2.2, 3.4, 1, 0.05, 0.03). A policy trained at the first and run at the second experiences a gap of five numbers. Saying which of the five matters is the experimental question.

One factor at a time

Let S(w) be the policy’s success probability in world w. The gap in success is S(nominal) − S(floor). To attribute it, change one factor at a time in two directions:

forward:   S(nominal with factor k set to its floor value)
backward:  S(floor with factor k set back to its nominal value)

If the factors acted independently, the two sweeps would give consistent answers and the losses would add. When the effect of one factor depends on the level of another (an interaction), they do not, and the sweeps disagree. The example below shows a case of exactly this.

A worked example

All numbers are real runs in Node against RLCore: RLCore.transferScore(θ, world, seed, 40) for seeds 1 to 5, so each rate is out of 200 episodes (the 200 starts are the first 40 from each seed’s starts(seed, 40, 'all'); a seed also fixes the noise stream). The policy is the one the nominal simulator picks, θ = {clear 0.3, kp 0, ff 1}. The intervals are 95% Wilson intervals on the pooled 200 episodes, which ignore that the five seeds are not independent draws of the world.

By hand: why the delay hurts and the gain does not

Command (1, 0) from rest. Nominal, dynStep gives v = 0.25 × 3 × 1 = 0.75 m/s after the first step, then 0.938, 0.984, 0.996, ... settling at 1.00; the position after four steps is 0.917 m. On the floor (gain 2.2, drag 3.4, delay 1) the first step has no effect (v = 0), then v = 0.25 × 2.2 = 0.55, 0.633, 0.645, 0.647 settling at 0.647 m/s; after four steps the robot has moved 0.457 m, half as far. Two distinct things happen. The speed is lower and the response is slower to build, and both are just time: the same path takes longer, which is what the green line shows. The delay is different: the policy computes each command from the position it sees now, and the command acts a quarter of a second later, when the robot is somewhere else. At the 0.65 to 1 m/s the robot moves at, that is 0.16 to 0.25 m. The nominal policy’s closest approach to the pillar centre over the 200 nominal episodes was 0.97 m at the minimum (median 1.09, 5th percentile 0.99), against contact at 0.95 m: a margin of 2 to 14 cm. A swerve that is late by a quarter metre has no margin to spend.

Decomposition, forward

Per seed, the full floor gave 0.675, 0.65, 0.725, 0.8 and 0.775 (the seed-to-seed spread of 0.15 is about the width of the pooled interval, so the pooled number is the fairer summary); delay alone gave 0.35, 0.425, 0.4, 0.575 and 0.45. Reading the table: the dynamics change (gain, drag) costs time, from 29.5 to 44.5 steps, and nothing else; process noise at 0.03 costs nothing visible; sensing noise at 0.05 costs 15 crashes in 200; the delay alone costs 112. All failures were crashes (no timeouts). Look at the last two columns together: the cheapest-looking change in the simulator, one more step of latency, is worth more than any other single factor.

Decomposition, backward, and the interaction

Removing the delay repairs everything (200 of 200), as the forward sweep said it should. But restoring the nominal gain lowers success from 145 to 100: with a one-step delay, a faster robot travels further during the delay, so the late swerve misses by more. Dynamics are harmless on their own and harmful in the presence of latency. That is an interaction, and it is why a table of "the effect of each factor" is not a budget that adds up. (Removing noise barely changes anything because with that set of factors the 55 floor crashes are dominated by the delay; the 138 versus 145 difference is within what five seeds of 40 starts can produce.)

Size of the gap in the one factor that matters

Setting only the delay to 0, 1, 2 and 3 steps, the same policy succeeded in 200, 88, 68 and 64 of 200 episodes, and with observation noise alone at 0, 0.05, 0.1 and 0.2 it succeeded in 200, 185, 168 and 140. The first curve falls off a cliff at the first step of delay; the second falls gradually. Robustness is not one number: the policy is fragile in one direction and sturdy in another, which is lesson M10-T4’s subject.

What this toy simplifies. The five parameters are the only things that differ; the structure of the dynamics is identical in both worlds, which is almost never true of a real robot. The policy has three numbers. There is no contact model. The "floor" is itself a simulator I wrote, so I know its parameters exactly, which no engineer does. The real lesson of the toy is the method (change one factor at a time, in both directions, with intervals), not the particular ranking.

Engineering implications and limitations

Practical rules for dealing with the gap

• Assume a gap exists and measure it. The question is how large and in which parts, not whether. Run the simulation-trained policy on the real robot (or the best proxy you have) in the safest staged way (lesson M10-T5), and log commands, states and outcomes.
• Decompose before you fix. Change one factor at a time in the simulator, forward and backward, and report each with an interval and several seeds (lesson M11-T2). A decomposition tells you whether to improve the model, randomize it, or add a layer such as the safety wrapper of lesson M9-T4.
• Check for margins the policy consumes. The nominal policy passed the pillar 2 to 14 cm from contact. A tight margin in the simulator is a warning, not a success: the simulator’s reward asked for it (shorter paths score better) and the real world will not honour it.
• Measure latency explicitly. Timestamp a command and the first visible effect. Latency is the gap factor that can be measured cheaply and hurts most, and it often hides inside "dynamics" (lesson M10-T3 shows that a fit that ignores it goes wrong).
• Do not trust one scalar. A policy can be sturdy against noise and brittle against delay. Report the curve, not one point (lesson M10-T4, lab M10-L2).
• Keep the simulator honest about what it leaves out. Write down the list of unmodelled effects (here: contact, friction, anything not in the five parameters). Those are where surprises will come from.
• Expect the gap to move. Wear, payload, temperature, a software update and a different floor each shift parameters. A result for one floor is a result for that floor on that day.

The cures, in one table

Common misconceptions

Engineering takeaways

1. The reality gap is the sum of many differences: dynamics, sensing, latency and contact are the four to name first; the toy includes the first three and says nothing about contact.
2. In this toy, the policy trained in the nominal world reached the goal in 200 of 200 nominal episodes and in 145 of 200 on the floor (95% interval 66% to 78%).
3. One factor at a time: the floor’s gain and drag alone cost time (29.5 to 44.5 steps) and no successes; observation noise cost 15 crashes; a single step of delay cost 112.
4. Factors interact: resetting the gain to its nominal value on the floor reduced success from 145 to 100, because delay and speed together set how far the robot travels before a command acts, so decompose in both directions.
5. A policy that exploits the simulator (here a pillar margin of 2 to 14 cm) scores perfectly and has nothing in reserve; the next lessons spend that reserve on randomization (M10-T2) and on identification (M10-T3).

References and further reading

The terms and the concerns of this lesson (reality gap, latency, unmodelled dynamics) are standard in the sim-to-real literature. The first two references are early works that targeted specific gap sources; all numbers above come from my own toy runs and are not results from these papers.

• J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba and P. Abbeel, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
• X. B. Peng, M. Andrychowicz, W. Zaremba and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization”, IEEE International Conference on Robotics and Automation (ICRA), 2018.
• J. Tan et al., “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots”, Robotics: Science and Systems (RSS), 2018.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Lesson M10-T2 tries the first cure: train over many simulated worlds instead of one. The central questions are which parameters to vary and over what ranges, what cautiousness costs in steps, and what happens when the ranges are too narrow to contain the floor or too wide to be useful.

Related content

Nothing below is required to finish this lesson.

• Lab
  M10-L1: Randomize the Simulator

  The policy was trained on one friction value: choose parameter ranges and see whether it survives the real floor.