M10-T2 — Domain Randomization

Learning objectives

After this lesson, you should be able to:

• Write the randomized training objective, sample a world from ranges by hand, and say which parameters of the toy are integers and why.
• Report the transfer of a randomized policy over several training seeds, with its cost in steps relative to the policy trained on one world.
• Show that ranges that do not contain the real parameters (too narrow or off-centre) fail, and that ranges that are far too wide cost time and make training noisier.
• State two things randomization cannot do: help outside its ranges, and tell you which parameter was the one that mattered.

The intuition

Look at the figure first. It plots success on the floor against the delay, with the policy held fixed. The orange policy is the one the nominal simulator picks (the tightest swerve, no velocity feedback): perfect with no delay, 145 of 200 at the floor’s delay of one step, 64 to 87 beyond. The blue policy is what the same training procedure picks when each training episode comes from a world with a random delay of 0, 1 or 2 steps (and random gain, drag and noise): it gives a wide swerve (extra clearance 0.9 m) with some velocity feedback. It succeeds in 200 of 200 episodes at delays 0, 1 and 2, the ones it saw, and in 197 at 3, which it never saw. It falls off at 4 and 5. The shaded band is the training range.

Three things are in that picture. The policy that was trained on many worlds is robust inside the band and a little beyond it, not forever. Cautiousness cost something (blue is more careful than orange at delay 0 too), though in this toy the price turns out to be small, about one step in thirty. And the green curve, a policy in between, shows that the choice is not binary: it is a family, and the training distribution selects a member.

The lesson is how to pick the band and what else to randomize, and how to tell whether it worked.

The mathematics

The randomized objective

Let w be a vector of world parameters, p(w) a distribution over it (here independent uniform draws over ranges), and J(θ, w) the performance of a policy with parameters θ in world w. Ordinary training maximises J(θ, w_sim) at one point. Domain randomization maximises an average over the distribution:

It is not "robust control" in the worst-case sense: an average over the distribution tolerates failures in a few worlds if the rest compensate, and it is the distribution, not the average itself, that decides which worlds count. A worst-case objective, min over w of J(θ, w), would favour the hardest world. I did not test it, and I use the average because that is what the toy’s training does.

The toy

ranges

A map from parameter names to [low, high], e.g. gain [2, 3.2], drag [2.8, 3.8], delay [0, 2], obsNoise [0, 0.08], noise [0, 0.04]. A parameter without a range keeps its nominal value.

sampleWorld

Draws every ranged parameter as low + (high − low) u with u uniform in [0, 1); the delay is an integer, low + floor(u (high − low + 1)), because the dynamics shift commands by whole steps.

trainTheta

For each θ in a grid of 15 policies (clear in {0.9, 0.7, 0.5, 0.3, 0.15} m times kp in {0, 1, 2}), runs the same episodes in freshly drawn worlds and keeps the θ with the best success − 0.01 × mean steps. With no ranges it trains on the nominal world only.

coverage

The real parameters should lie inside the ranges, ideally with room beyond them for what you did not measure.

Choosing what to randomize

Choose the parameters from what you know about the gap (lesson M10-T1): the ones the simulator gets wrong, in the units you can bound, with ranges at least as wide as your uncertainty about the real value. The usual candidates are the dynamics (masses, friction, gains), the sensing (noise, delay, bias), the actuation (latency, strength) and, for vision, the appearance. The parameters should be physically sensible: a world with a gain of zero is not a harder version of the task, it is an impossible one.

The cost of cautiousness

A policy that works in many worlds is typically one that does not use the margin that works in one. In the toy, that is the extra clearance clear: 0.3 m is a tight swerve, 0.9 m a wide one. The wide swerve is a longer path (more steps) and the simulator’s score counts steps, so the nominal simulator prefers the tight one by a small margin, as the example shows, while the randomized training prefers the wide one by a large margin.

A worked example

Every number below is a real run in Node against RLCore. Training: trainTheta(ranges, seed, 30) for seeds 1 to 5 (30 episodes per θ per seed). Evaluation: each trained θ on transferScore(θ, world, 100 + seed, 40), 5 seeds x 40 starts = 200 episodes, on the floor (gain 2.2, drag 3.4, delay 1, obsNoise 0.05, noise 0.03), and on a "hard" world outside the floor (gain 1.8, drag 3.8, delay 2, obsNoise 0.08, noise 0.04). "Full" ranges are the five above. Steps are the mean over successful episodes (so a policy that crashes early does not look faster).

By hand: sampling a world and scoring two policies

With the full ranges and uniforms (0.5, 0.25, 0.9, 0.5, 0.1) in the order gain, drag, delay, obsNoise, noise: gain = 2 + 1.2 × 0.5 = 2.6; drag = 2.8 + 1.0 × 0.25 = 3.05; delay = 0 + floor(0.9 × 3) = 2; obsNoise = 0.08 × 0.5 = 0.04; noise = 0.04 × 0.1 = 0.004. That is one training world. Now the scores that trainTheta computed for seed 1. In the nominal simulator, θ = {0.9, 1} succeeds in every episode and takes 31.8 steps on average, scoring 1 − 0.01 × 31.8 = 0.682; θ = {0.3, 0} succeeds in every episode in 30.4 steps, scoring 0.696. The nominal simulator therefore picks the tight policy, by 0.014: it pays 1.4 steps for 0.6 m of clearance. Over the randomized worlds, {0.9, 1} succeeds in all episodes in 40.5 steps (0.595) and {0.3, 0} in 53% of episodes in 38.6 steps (0.147). The choice flips, by 0.448.

Which ranges

Reading it. The ranges that do not contain the floor fail to a degree set by how far the floor is from them (161 of 200 for ±10% around nominal, which excludes the floor’s gain of 2.2 and its delay; 175 for a ±5% band that excludes the delay and the floor’s gain). The "full" ranges, which contain the floor’s values with room around them, reach the oracle: 200 of 200 on the floor, 199 on a world outside the training ranges in its gain, 1.8 (the hard world). Making them twice as wide cost no successes and about 2.6 steps (46.1 against 43.5); a six-step delay and a gain going down to 0.8 cost 3.3 steps (46.8). In the nominal world all randomized policies took 30.2 to 30.4 steps against 28.9 for the nominal-trained one: cautiousness costs 1.3 to 1.5 steps of 29, about 5%.

One of those steps numbers needs separating from the other. The floor takes 43.5 steps against 28.9 in the nominal world for the same policy family, but that is mainly the floor’s lower gain and higher drag (the tight policy, which fails 31% of the floor episodes, takes 45.3 steps when it succeeds), not the extra clearance. The price of randomization is the 1.3 to 1.5 steps, not the 15.

Which parameters

Every subset moved the policy away from the tight one and nearly every subset got the floor right. That is a lesson and a warning. The lesson: randomizing anything that makes the simulator harder pushes the grid toward more clearance, and more clearance is what the delay needs. The warning: a floor success of 199 or 200 does not say whether the right parameter was randomized. The hard world, which has a longer delay and more noise than the floor, separates them a little (gain and drag only: 183 of 200; delay: 195 to 199), and the lab asks you to find such tests yourself.

Beyond the ranges

For the floor with only the delay changed, the policy trained on delays 0 to 2 (the blue curve in the figure) had 200, 200, 200 successes of 200 at delays 0, 1, 2, then 197, 152 and 91 at 3, 4 and 5. The tight policy had 200, 145, 87, 69, 65, 64. The ranges are a promise, with a little headroom. The 10-seed check of the "full" ranges on a floor with a delay of 3 steps gave 375 of 400 with 30 training episodes and 395 of 400 with 100: the same ranges, but a noisier choice (two of ten 30-episode runs picked {0.7, 1}, none of the 100-episode runs did).

When randomization hurts

I found three ways. (1) Wide ranges cost time. 2.6 to 3.3 steps (6% to 8%) on the floor, as in the table, for no gain on the floor. (2) Ranges that are wide in the wrong way make the choice noisier. With delays up to 8 steps, or a gain that goes down to 0.3, the training worlds contain cases where nothing succeeds, so they add noise to the score and little signal. On the floor with a delay of 3 steps those two settings gave 179 and 175 of 200, against 188 for the full ranges (10-seed figure scaled to 200), with picks such as {0.7, 0} and {0.9, 0} that the full ranges never made. With five seeds the intervals overlap (the 175 has an interval of 82% to 91%), so I call this suggestive, not shown. (3) Ranges that exclude the truth (rows 2 and 3 of the first table) give a false sense of robustness: the policy is trained on many worlds and none is the real one. I did not test cases where randomization of an irrelevant parameter wastes capacity, because the policy has three numbers and cannot waste any.

What the toy simplifies. The policy family is three numbers chosen from a grid of 15 by exhaustive search, so "training" is selection, not learning, and the policy cannot learn to infer the world (adapt) as a recurrent policy can. The cost of cautiousness is small because the extra clearance costs only a short detour in an open hall; in a corridor it would not be available. The training ranges and the floor were chosen by me knowing the floor.

Engineering implications and limitations

Practical rules

• Start from the decomposition. Randomize the parameters that lesson M10-T1’s one-factor tests showed matter, and the ones you cannot measure. In the toy, the delay is the one a policy tolerates least; randomizing it was the cheapest route to the floor.
• Make the ranges cover the truth with room. Centre them on the best estimate of the real value, not on the simulator’s default, and widen them by your uncertainty plus a margin. Ranges around the default failed (161 and 175 of 200); ranges that contained the floor did not.
• Do not go much wider than you need. Each extra step of range has a price in time and in training signal, and a gain near zero or a delay of 8 are not the real robot. Widen them only to cover things you cannot rule out.
• Pay for the extra worlds. Randomized training needs more episodes: in the toy, 30 episodes per policy picked the clearance 0.9 in 8 of 10 seeds, 100 episodes in 10 of 10. For real learning the cost is simulation throughput and training time.
• Test outside the ranges. A held-out world with a larger delay or noise than the training ranges (the hard world) shows how much headroom there is; lesson M10-T4 and lab M10-L2 turn this into robustness curves.
• Keep a log of what you randomized. A policy is only as trustworthy as the list of variations it was exposed to (lesson M11-T2 on reporting seeds).
• Consider automating the widening. OpenAI’s Rubik’s-cube work increased the ranges automatically while the policy kept succeeding (automatic domain randomization). I describe it in words; I did not implement it.
• Combine with other defences. Randomization makes the policy tolerant; it does not make it safe. The wrapper of lesson M9-T4 and the staged rollout of lesson M10-T5 are still needed.

Randomize, or identify?

If you can measure the real parameters (gain, drag and delay can be identified from a log, lesson M7-T2), randomizing around the measured values with a modest width uses the information you have. If you cannot, wide ranges are what remains. Lesson M10-T3 compares the two, and the combination, on the same floor.

Common misconceptions

Engineering takeaways

1. Domain randomization trains on an average over worlds drawn from ranges; the ranges encode what you believe about the real world, so they must contain it.
2. Nominal-only training picked the tight policy {clear 0.3, kp 0} in all five seeds and reached the goal in 139 of 200 floor episodes; the full ranges reached 200 of 200, the same as training on the exact floor.
3. The price of cautiousness in this toy is small: 1.3 to 1.5 steps in the nominal world, 2.6 to 3.3 more for ranges much wider than needed; ranges that exclude the floor cost much more (161 to 175 of 200).
4. Randomization is robust inside its ranges and some way beyond (delay 3: 197 of 200) and then it ends (delay 5: 91); it also does not say which parameter mattered, because randomizing almost any subset gave a cautious policy.
5. The toy is a grid search over three numbers, with ranges chosen by someone who knew the floor; use it to learn the shape of the trade-offs, not the sizes.

References and further reading

The three works below introduced or popularised randomization for transfer: visual randomization, dynamics randomization, and automatic widening of the ranges. They are the primary sources for the ideas I describe; none of my numbers come from them.

• J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba and P. Abbeel, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
• X. B. Peng, M. Andrychowicz, W. Zaremba and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization”, IEEE International Conference on Robotics and Automation (ICRA), 2018.
• OpenAI (I. Akkaya et al.), “Solving Rubik’s Cube with a Robot Hand”, arXiv:1910.07113, 2019.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Lesson M10-T3 asks the opposite question: instead of covering the unknown with ranges, measure it. It fits the floor’s gain, drag and delay from a log with the least-squares tools of lesson M7-T2, trains in the calibrated world, and compares calibrating, randomizing and doing both, honestly, across seeds.

Related content

Nothing below is required to finish this lesson.

• Before
  M10-T1: The Reality Gap

  Name the sources of difference between simulation and reality (dynamics, sensing, latency, contact) and how each shows up.

• Lab
  M10-L1: Randomize the Simulator

  The policy was trained on one friction value: choose parameter ranges and see whether it survives the real floor.