Domain randomization for a walking robot: does it stop the falls?
Last week, after we fixed our walking robot's score, it still tipped over too often from a slightly different start: PPO 23 of 60 runs, CMA-ES 15 of 60. This week we tried the standard cure, domain randomization: every training trial gets its own random floor friction and start pose. It worked for one method and not the other. CMA-ES tipped 5 times out of 60 instead of 15, and walked more than twice as far. PPO tipped 24 times instead of 23: no better.
CMA-ES seed 3, same start pose (every joint nudged by up to 3°), default floor. Left: trained on one world, it tips at 1.46 s. Right: trained on random worlds, it stays level and covers 8.41 m in 5 s. Physics simulation (MuJoCo), not video footage.
What we randomized
The robot, the two methods and the score are the same as last week: a 36 cm box with four two-joint legs in MuJoCo [1], 5-second trials, and the strict rule that a trial that tilts past 30° scores −1 m. CMA-ES [3] evolves an open-loop sine gait (25 numbers); PPO [2] learns a neural-network policy with Stable-Baselines3 defaults [4].
Domain randomization means training on many slightly different simulations so the controller can't rely on one exact world. It was first used for camera images [5], then for robot dynamics [6]; Google's Minitaur quadruped randomized contact friction from 0.5 to 1.25, along with nine other values [7]. We kept it small, two things:
- Floor friction from 0.5 to 1.5 (default 1.0), drawn fresh for each trial.
- Start pose: every joint angle nudged by up to ±0.05 rad (about 3°), the same kind of nudge last week's test used.
def model_with_friction(f):
"""A copy of the robot whose sliding friction (all geoms, so every foot-floor contact) is f."""
m = copy.deepcopy(MODEL)
m.geom_friction[:, 0] = f
return m
def randomize_start(model, d, rng):
"""Random start pose (only joint angles; the torso still starts level at 0.3 m)."""
d.qpos[7:] += rng.uniform(-JITTER_RAD, JITTER_RAD, d.qpos[7:].shape)
mujoco.mj_forward(model, d)
def random_world(rng):
"""One randomized training trial: (model with random friction, MjData with random start pose)."""
m = model_with_friction(rng.uniform(*FRICTION_RANGE))
d = mujoco.MjData(m)
randomize_start(m, d, rng)
return m, d
PPO gets a new random world at every episode reset. CMA-ES gives each of its 24 candidates one trial in its own random world. The budget is unchanged: 12,000 five-second trials (15 million physics steps), seeds 1–3. One more change for CMA-ES: with a noisy score, the "best ever" candidate is just the luckiest one, so the randomized run keeps the search mean as its final gait. To check that this choice isn't what helped, we ran a control: CMA-ES on the fixed world, keeping the mean the same way.
Results
Every final controller was judged by the strict rule, replayed for the full 5 s: first last week's test (default floor, 20 random start poses each), then a friction sweep with 10 random start poses at each of seven friction values.
| Method | Trained on | Strict score, 20 random starts (mean of 3 seeds) | Runs that tipped | Tipped in the friction sweep, 0.5–1.5 | Training CPU per run |
|---|---|---|---|---|---|
| CMA-ES | One world (last week) | 2.81 m | 15 / 60 | 71 / 150 | 214 s |
| CMA-ES | One world, keep mean (control) | 2.86 m | 14 / 60 | 83 / 150 | 221 s |
| CMA-ES | Random worlds | 6.85 m | 5 / 60 | 54 / 150 | 190 s |
| PPO | One world (last week) | 4.02 m | 23 / 60 | 77 / 150 | 1830 s |
| PPO | Random worlds | 3.63 m | 24 / 60 | 72 / 150 | 1857 s |
A tipped run scores −1. The "one world" rows are last week's training re-run from scratch: same seeds, same numbers to the last digit. CPU time is process time on one core of a cloud Linux machine.

Runs that tipped, out of 30 (3 seeds × 10 start poses), by floor friction. * = outside the training range.
| Friction | 0.3* | 0.5 | 0.75 | 1 | 1.25 | 1.5 | 2* |
|---|---|---|---|---|---|---|---|
| CMA-ES, one world | 28 | 21 | 9 | 8 | 15 | 18 | 17 |
| CMA-ES, random worlds | 19 | 20 | 10 | 1 | 5 | 18 | 26 |
| PPO, one world | 24 | 15 | 14 | 11 | 17 | 20 | 19 |
| PPO, random worlds | 25 | 12 | 11 | 14 | 20 | 15 | 14 |

What we learned
- For CMA-ES, randomization did the job. Tips from random starts fell from 15 to 5 of 60 and the strict score went from 2.8 m to 6.8 m. The control run (14 of 60, 2.9 m) shows the gain came from the random worlds, not from keeping the search mean. The open-loop gait can't react to anything, so the only way it survives a nudged start is a rhythm that tolerates one, and random worlds are what select for that.
- It even walked farther on the default floor. The randomized CMA-ES gaits covered 6.09, 8.27, 8.37 m from the normal start, against 4.42, 4.64, 6.05 m before. Our guess: the noise kept the search from settling early on a gait that only works on one exact world.
- For PPO, it made no clear difference. 23 vs 24 tips from 60 starts, and close in the sweep (77 vs 72 of 150 inside the range). Its failures look like a different problem: its checkpoints after 2,400 trials scored −1 92 of 300 times without randomization and 102 of 300 with it. The policy keeps changing, and some versions fall whatever the floor is.
- Randomization only covers what you randomize. At friction 2.0, never seen in training, the randomized CMA-ES gaits tipped 26 of 30 times, against 17 for the one-world gaits. At 0.3 they did better (19 vs 28), but nobody walked well on a floor that slippery.
- The edges of the range are still hard. Even inside training, at friction 0.5 and 1.5 the randomized gaits tipped about as often as before (20 vs 21 and 18 vs 18 of 30). One trial per candidate in a random world rewards gaits that do well on average, and most of the range is near the middle.
Ways to make a sim-trained walker robust, compared
| Approach | What it does | Cost / risk | Tested here |
|---|---|---|---|
| Domain randomization | Random physics and start states during training | Free in compute (same budget); only covers what you randomize; ranges are guesses | Yes: helped CMA-ES, not PPO |
| Keep the best checkpoint | Ship the policy version that scored best on many test starts | Needs a test set; the test must look like deployment | No (next candidate for PPO) |
| Feedback control | Let the controller see tilt and correct it (PPO already does) | Needs sensors and a policy that uses them well | Partly: PPO has it, CMA-ES doesn't |
| Measure the real values | System identification: measure friction, mass, motor strength and set the sim to match | Needs the real robot and test rigs | No (planned when our Kit 1 samples arrive) |
| Wider ranges or more variables | Mass, motor strength, latency, sensor noise [7] | Too wide and the task becomes too hard to learn | No |
Run it yourself
The download has the exact code we ran, plus the analysis, figure and video scripts. From a fresh virtual environment (Python 3.13.18, mujoco 3.16.0, stable-baselines3 2.9.0, torch 2.14.1, cma 4.5.0):
$ python -m venv venv && source venv/bin/activate $ pip install -r requirements.txt $ python cmaes_walker.py gen 10 trials 240 champion 0.15 m cpu 1.8 s gen 20 trials 480 champion 0.14 m cpu 5.0 s gen 30 trials 720 champion -1.00 m cpu 8.4 s gen 40 trials 960 champion 1.17 m cpu 12.2 s gen 50 trials 1200 champion 1.60 m cpu 16.6 s gen 60 trials 1440 champion 2.18 m cpu 20.6 s gen 70 trials 1680 champion 3.26 m cpu 24.9 s gen 80 trials 1920 champion 3.61 m cpu 29.0 s gen 90 trials 2160 champion 3.14 m cpu 33.1 s gen 100 trials 2400 champion 3.93 m cpu 37.2 s
$ python ppo_walker.py physics steps 120000 trials 96 policy 0.04 m cpu 14.3 s physics steps 240000 trials 192 policy 0.08 m cpu 29.6 s physics steps 360000 trials 288 policy 0.09 m cpu 43.9 s physics steps 480000 trials 384 policy 0.34 m cpu 57.5 s physics steps 600000 trials 480 policy 1.13 m cpu 71.2 s physics steps 720000 trials 576 policy -1.00 m cpu 85.1 s physics steps 840000 trials 672 policy 3.74 m cpu 98.6 s physics steps 960000 trials 768 policy 5.28 m cpu 111.7 s physics steps 1080000 trials 864 policy 5.29 m cpu 125.0 s physics steps 1200000 trials 960 policy 5.54 m cpu 139.0 s physics steps 1320000 trials 1056 policy 6.72 m cpu 152.1 s physics steps 1440000 trials 1152 policy 7.55 m cpu 164.5 s physics steps 1560000 trials 1248 policy 6.37 m cpu 178.2 s physics steps 1680000 trials 1344 policy 6.60 m cpu 191.3 s physics steps 1800000 trials 1440 policy 7.12 m cpu 204.9 s physics steps 1920000 trials 1536 policy 7.25 m cpu 218.8 s physics steps 2040000 trials 1632 policy 8.17 m cpu 232.2 s physics steps 2160000 trials 1728 policy 7.40 m cpu 245.0 s physics steps 2280000 trials 1824 policy 8.25 m cpu 258.5 s physics steps 2400000 trials 1920 policy 7.91 m cpu 272.3 s physics steps 2520000 trials 2016 policy 8.47 m cpu 286.3 s physics steps 2640000 trials 2112 policy 9.85 m cpu 299.2 s physics steps 2760000 trials 2208 policy 9.80 m cpu 312.9 s physics steps 2880000 trials 2304 policy -1.00 m cpu 326.3 s physics steps 3000000 trials 2400 policy 8.56 m cpu 338.7 s
That is real output, with cma's one-line "matplotlib not installed" warning removed. The defaults are random worlds, seed 1, 2,400 trials; these lines match the first 2,400 trials of our full runs exactly, because the physics and the random draws are seeded. Note the −1 lines: the policy or gait being logged tipped on that check. For the page's runs: python cmaes_walker.py 1 500 random and python ppo_walker.py 1 3000000 random; use fixed for last week's training. CPU seconds depend on your machine. No GPU needed.
Download domain-randomization-walker.zip or the single files: walker_env.py, cmaes_walker.py, ppo_walker.py, requirements.txt
Limits
- Three seeds per setup, and seeds differ a lot. Treat the averages as a rough picture, especially the friction sweep (30 runs per point).
- We randomized only friction and start pose. Mass, motor strength, delay and sensor noise were fixed.
- PPO used Stable-Baselines3 defaults and its final policy, not its best checkpoint. Tuning either would change its numbers.
- Simulation only. Friction 1.0 is MuJoCo's default, not a measured rubber-on-floor value.
Evolve your own walkers
Robot Evolution Lab is the full CMA-ES code behind our videos: 2 to 8 legs, stairs and ice, body-shape evolution and a video recorder.
See Robot Evolution LabGet the next robot experiment
One email when we publish a new experiment, about once a week. Unsubscribe anytime.
FAQ
What is domain randomization?
Training on many slightly different versions of the simulation instead of one, so the controller can't depend on exact values that the real world won't match. Tobin et al. (2017) randomized rendering for vision; Peng et al. (2018) randomized dynamics such as mass and friction for robot control.
Did domain randomization make our walking robot more robust?
For CMA-ES, yes: from 20 random start poses on the default floor, its gaits tipped 15 of 60 times before and 5 of 60 after, and the strict score rose from 2.8 m to 6.8 m. For PPO, no: 23 of 60 before, 24 of 60 after.
Why didn't domain randomization help PPO here?
We don't know for sure. Our best guess: PPO's tips mostly come from its policy changing with every update, not from the world. Its checkpoints after 2,400 trials scored −1 92 of 300 times without randomization and 102 of 300 times with it. A tuned PPO, more training or keeping the best checkpoint may behave differently.
Does a robot trained with domain randomization work outside the range it saw?
Not in our test. CMA-ES trained on friction 0.5 to 1.5 tipped in 26 of 30 runs at friction 2.0, worse than the version trained only at 1.0 (17 of 30). Randomize over the whole range the real robot might meet.
How wide should the friction range be?
Wide enough to cover the real surfaces. Tan et al. (2018) randomized contact friction from 0.5 to 1.25 for the Minitaur quadruped, the typical range between its rubber feet and carpet. We used 0.5 to 1.5 around MuJoCo's default of 1.
Sources
- E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
- N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
- A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
- J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, P. Abbeel, Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, arXiv:1703.06907, 2017.
- X. B. Peng, M. Andrychowicz, W. Zaremba, P. Abbeel, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, ICRA 2018 (arXiv:1710.06537).
- J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, V. Vanhoucke, Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, arXiv:1804.10332, 2018 (Table I: randomized parameters, contact friction 0.5–1.25).
- Experiment, code and measurements: TinyBot Forge, November 2026. Every number on this page comes from the code in the download.
More: Reward hacking: fixing the score · CMA-ES vs PPO, the first test · all guides