TINYBOT FORGE
Home › Learn › Domain randomization

Domain randomization for a walking robot: does it stop the falls?

Our experimentMuJoCo + Python15 new training runsUpdated November 16, 2026

Last week, after we fixed our walking robot's score, it still tipped over too often from a slightly different start: PPO 23 of 60 runs, CMA-ES 15 of 60. This week we tried the standard cure, domain randomization: every training trial gets its own random floor friction and start pose. It worked for one method and not the other. CMA-ES tipped 5 times out of 60 instead of 15, and walked more than twice as far. PPO tipped 24 times instead of 23: no better.

CMA-ES seed 3, same start pose (every joint nudged by up to 3°), default floor. Left: trained on one world, it tips at 1.46 s. Right: trained on random worlds, it stays level and covers 8.41 m in 5 s. Physics simulation (MuJoCo), not video footage.

15 to 5 / 60CMA-ES runs that tipped, 20 random starts × 3 seeds, before and after randomization
23 to 24 / 60PPO runs that tipped, same test: no real change
17 to 26 / 30CMA-ES tips at friction 2.0, outside the training range: worse

What we randomized

The robot, the two methods and the score are the same as last week: a 36 cm box with four two-joint legs in MuJoCo [1], 5-second trials, and the strict rule that a trial that tilts past 30° scores −1 m. CMA-ES [3] evolves an open-loop sine gait (25 numbers); PPO [2] learns a neural-network policy with Stable-Baselines3 defaults [4].

Domain randomization means training on many slightly different simulations so the controller can't rely on one exact world. It was first used for camera images [5], then for robot dynamics [6]; Google's Minitaur quadruped randomized contact friction from 0.5 to 1.25, along with nine other values [7]. We kept it small, two things:

def model_with_friction(f):
    """A copy of the robot whose sliding friction (all geoms, so every foot-floor contact) is f."""
    m = copy.deepcopy(MODEL)
    m.geom_friction[:, 0] = f
    return m

def randomize_start(model, d, rng):
    """Random start pose (only joint angles; the torso still starts level at 0.3 m)."""
    d.qpos[7:] += rng.uniform(-JITTER_RAD, JITTER_RAD, d.qpos[7:].shape)
    mujoco.mj_forward(model, d)

def random_world(rng):
    """One randomized training trial: (model with random friction, MjData with random start pose)."""
    m = model_with_friction(rng.uniform(*FRICTION_RANGE))
    d = mujoco.MjData(m)
    randomize_start(m, d, rng)
    return m, d

PPO gets a new random world at every episode reset. CMA-ES gives each of its 24 candidates one trial in its own random world. The budget is unchanged: 12,000 five-second trials (15 million physics steps), seeds 1–3. One more change for CMA-ES: with a noisy score, the "best ever" candidate is just the luckiest one, so the randomized run keeps the search mean as its final gait. To check that this choice isn't what helped, we ran a control: CMA-ES on the fixed world, keeping the mean the same way.

Results

Every final controller was judged by the strict rule, replayed for the full 5 s: first last week's test (default floor, 20 random start poses each), then a friction sweep with 10 random start poses at each of seven friction values.

MethodTrained onStrict score, 20 random starts (mean of 3 seeds)Runs that tippedTipped in the friction sweep, 0.5–1.5Training CPU per run
CMA-ESOne world (last week)2.81 m15 / 6071 / 150214 s
CMA-ESOne world, keep mean (control)2.86 m14 / 6083 / 150221 s
CMA-ESRandom worlds6.85 m5 / 6054 / 150190 s
PPOOne world (last week)4.02 m23 / 6077 / 1501830 s
PPORandom worlds3.63 m24 / 6072 / 1501857 s

A tipped run scores −1. The "one world" rows are last week's training re-run from scratch: same seeds, same numbers to the last digit. CPU time is process time on one core of a cloud Linux machine.

Two line charts against floor friction from 0.3 to 2. CMA-ES trained on random worlds tips least at friction 1 and 1.25 and scores about 7 m at friction 1, but tips most at friction 2. The PPO lines with and without randomization stay close together.
Friction sweep: every final controller from 10 random start poses at each friction, 3 seeds. Shaded: the friction range used in randomized training.

Runs that tipped, out of 30 (3 seeds × 10 start poses), by floor friction. * = outside the training range.

Friction0.3*0.50.7511.251.52*
CMA-ES, one world282198151817
CMA-ES, random worlds192010151826
PPO, one world24151411172019
PPO, random worlds25121114201514
Learning curves. CMA-ES on random worlds climbs to about 7.5 m on the default robot; on one world it levels off near 5 m. The two PPO curves look alike and jump between 8 m and -1 m.
Learning curves, scored on the default robot. CMA-ES: the gait it would ship at that moment. PPO: its current policy, one trial every 96 trials. Mean of 3 seeds.

What we learned

Ways to make a sim-trained walker robust, compared

ApproachWhat it doesCost / riskTested here
Domain randomizationRandom physics and start states during trainingFree in compute (same budget); only covers what you randomize; ranges are guessesYes: helped CMA-ES, not PPO
Keep the best checkpointShip the policy version that scored best on many test startsNeeds a test set; the test must look like deploymentNo (next candidate for PPO)
Feedback controlLet the controller see tilt and correct it (PPO already does)Needs sensors and a policy that uses them wellPartly: PPO has it, CMA-ES doesn't
Measure the real valuesSystem identification: measure friction, mass, motor strength and set the sim to matchNeeds the real robot and test rigsNo (planned when our Kit 1 samples arrive)
Wider ranges or more variablesMass, motor strength, latency, sensor noise [7]Too wide and the task becomes too hard to learnNo

Run it yourself

The download has the exact code we ran, plus the analysis, figure and video scripts. From a fresh virtual environment (Python 3.13.18, mujoco 3.16.0, stable-baselines3 2.9.0, torch 2.14.1, cma 4.5.0):

$ python -m venv venv && source venv/bin/activate
$ pip install -r requirements.txt
$ python cmaes_walker.py
gen  10  trials   240  champion  0.15 m  cpu    1.8 s
gen  20  trials   480  champion  0.14 m  cpu    5.0 s
gen  30  trials   720  champion -1.00 m  cpu    8.4 s
gen  40  trials   960  champion  1.17 m  cpu   12.2 s
gen  50  trials  1200  champion  1.60 m  cpu   16.6 s
gen  60  trials  1440  champion  2.18 m  cpu   20.6 s
gen  70  trials  1680  champion  3.26 m  cpu   24.9 s
gen  80  trials  1920  champion  3.61 m  cpu   29.0 s
gen  90  trials  2160  champion  3.14 m  cpu   33.1 s
gen 100  trials  2400  champion  3.93 m  cpu   37.2 s
$ python ppo_walker.py
physics steps   120000  trials    96  policy  0.04 m  cpu   14.3 s
physics steps   240000  trials   192  policy  0.08 m  cpu   29.6 s
physics steps   360000  trials   288  policy  0.09 m  cpu   43.9 s
physics steps   480000  trials   384  policy  0.34 m  cpu   57.5 s
physics steps   600000  trials   480  policy  1.13 m  cpu   71.2 s
physics steps   720000  trials   576  policy -1.00 m  cpu   85.1 s
physics steps   840000  trials   672  policy  3.74 m  cpu   98.6 s
physics steps   960000  trials   768  policy  5.28 m  cpu  111.7 s
physics steps  1080000  trials   864  policy  5.29 m  cpu  125.0 s
physics steps  1200000  trials   960  policy  5.54 m  cpu  139.0 s
physics steps  1320000  trials  1056  policy  6.72 m  cpu  152.1 s
physics steps  1440000  trials  1152  policy  7.55 m  cpu  164.5 s
physics steps  1560000  trials  1248  policy  6.37 m  cpu  178.2 s
physics steps  1680000  trials  1344  policy  6.60 m  cpu  191.3 s
physics steps  1800000  trials  1440  policy  7.12 m  cpu  204.9 s
physics steps  1920000  trials  1536  policy  7.25 m  cpu  218.8 s
physics steps  2040000  trials  1632  policy  8.17 m  cpu  232.2 s
physics steps  2160000  trials  1728  policy  7.40 m  cpu  245.0 s
physics steps  2280000  trials  1824  policy  8.25 m  cpu  258.5 s
physics steps  2400000  trials  1920  policy  7.91 m  cpu  272.3 s
physics steps  2520000  trials  2016  policy  8.47 m  cpu  286.3 s
physics steps  2640000  trials  2112  policy  9.85 m  cpu  299.2 s
physics steps  2760000  trials  2208  policy  9.80 m  cpu  312.9 s
physics steps  2880000  trials  2304  policy -1.00 m  cpu  326.3 s
physics steps  3000000  trials  2400  policy  8.56 m  cpu  338.7 s

That is real output, with cma's one-line "matplotlib not installed" warning removed. The defaults are random worlds, seed 1, 2,400 trials; these lines match the first 2,400 trials of our full runs exactly, because the physics and the random draws are seeded. Note the −1 lines: the policy or gait being logged tipped on that check. For the page's runs: python cmaes_walker.py 1 500 random and python ppo_walker.py 1 3000000 random; use fixed for last week's training. CPU seconds depend on your machine. No GPU needed.

Download domain-randomization-walker.zip or the single files: walker_env.py, cmaes_walker.py, ppo_walker.py, requirements.txt

Limits

Evolve your own walkers

Robot Evolution Lab is the full CMA-ES code behind our videos: 2 to 8 legs, stairs and ice, body-shape evolution and a video recorder.

See Robot Evolution Lab

Get the next robot experiment

One email when we publish a new experiment, about once a week. Unsubscribe anytime.

Free. No spam, one email per experiment.

FAQ

What is domain randomization?

Training on many slightly different versions of the simulation instead of one, so the controller can't depend on exact values that the real world won't match. Tobin et al. (2017) randomized rendering for vision; Peng et al. (2018) randomized dynamics such as mass and friction for robot control.

Did domain randomization make our walking robot more robust?

For CMA-ES, yes: from 20 random start poses on the default floor, its gaits tipped 15 of 60 times before and 5 of 60 after, and the strict score rose from 2.8 m to 6.8 m. For PPO, no: 23 of 60 before, 24 of 60 after.

Why didn't domain randomization help PPO here?

We don't know for sure. Our best guess: PPO's tips mostly come from its policy changing with every update, not from the world. Its checkpoints after 2,400 trials scored −1 92 of 300 times without randomization and 102 of 300 times with it. A tuned PPO, more training or keeping the best checkpoint may behave differently.

Does a robot trained with domain randomization work outside the range it saw?

Not in our test. CMA-ES trained on friction 0.5 to 1.5 tipped in 26 of 30 runs at friction 2.0, worse than the version trained only at 1.0 (17 of 30). Randomize over the whole range the real robot might meet.

How wide should the friction range be?

Wide enough to cover the real surfaces. Tan et al. (2018) randomized contact friction from 0.5 to 1.25 for the Minitaur quadruped, the typical range between its rubber feet and carpet. We used 0.5 to 1.5 around MuJoCo's default of 1.

Sources

  1. E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
  2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
  3. N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
  4. A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
  5. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, P. Abbeel, Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, arXiv:1703.06907, 2017.
  6. X. B. Peng, M. Andrychowicz, W. Zaremba, P. Abbeel, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, ICRA 2018 (arXiv:1710.06537).
  7. J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, V. Vanhoucke, Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, arXiv:1804.10332, 2018 (Table I: randomized parameters, contact friction 0.5–1.25).
  8. Experiment, code and measurements: TinyBot Forge, November 2026. Every number on this page comes from the code in the download.

More: Reward hacking: fixing the score · CMA-ES vs PPO, the first test · all guides