CMA-ES vs reinforcement learning (PPO) for a walking robot: we tested both
Should you evolve a walking gait with CMA-ES or train it with reinforcement learning? We gave both the same four-legged robot, the same 5-second test and the same number of tries. With 2,400 trials, CMA-ES walked 4.2 m on average and PPO 1.8 m, and CMA-ES needed about 6× less CPU time. Then the rest of the story got more interesting.
Seed 1 of each method after 12,000 trials, replayed in real time. Physics simulation (MuJoCo), not video footage. The PPO robot learned to tip its body up on end and lunge forward; more on that below.
The setup: one robot, one test, two brains
The robot is the same one from our guide make a robot teach itself to walk: a 36 cm box torso, four legs, a hip and a knee motor on each leg, simulated in MuJoCo. Every trial lasts 5 simulated seconds (1,250 physics steps). The score is the distance walked forward, minus 1 m if the robot ends upside down. Both methods are optimizing exactly this number.
| CMA-ES | PPO (reinforcement learning) | |
|---|---|---|
| Controller | Each motor follows a sine wave. 8 amplitudes + 8 phases + 8 offsets + 1 frequency = 25 numbers | Neural network (2 hidden layers of 64) maps 24 sensor values to 8 motor targets, 50 times a second |
| Sees sensors? | No, it plays the same rhythm blind (open loop) | Yes: joint angles and speeds, torso tilt, speed and height (closed loop) |
| Learns from | One number per trial: the final score | The reward at every step (forward progress), plus a learned value estimate |
| Algorithm | CMA-ES [2][3], the cma package, 24 trials per generation | PPO [4], Stable-Baselines3 [5] default settings, 4 parallel environments of 512 steps (2,048 per update), normalized observations and rewards |
| Budget | Same physics steps for both. The main comparison is at 3 million steps (2,400 trials, CMA-ES's 100-generation default); we kept both running to 15 million (12,000 trials). Seeds 1, 2 and 3. | |
This is a deliberately practical comparison: each method as a hobbyist would use it, with its usual controller and default settings, on an ordinary CPU. It is not a tuned benchmark.
Update: we fixed the loophole
We fixed this score twice and re-ran both methods. The first fix bred a new hack; with the second, PPO walked farther than CMA-ES.
See the follow-upResults

| Same robot, same test | CMA-ES | PPO |
|---|---|---|
| Distance after 2,400 trials (mean of 3) | 4.16 m | 1.77 m |
| CPU time for 2,400 trials | 63 s | 376 s |
| Trials until it first walks 1 m (median) | 144 | 2,016 |
| Trials until 2 m | 480 | 2,496 |
| Trials until 3 m | 960 | 3,168 |
| Trials until 4 m | 1,728 | 4,800 |
| Distance after 12,000 trials (mean) | 5.69 m | 2.36 m |
| Best single evaluation, any time (mean) | 5.69 m | 4.59 m |
CMA-ES is logged every generation (24 trials), PPO every 96 trials, so PPO's "trials until" numbers are rounded up to the next 96. "Distance after 12,000 trials" is the controller saved at the end of the run (for PPO that is after the last update, so it can differ from the curve's last point).
| Per seed | CMA-ES @2,400 | PPO @2,400 | CMA-ES @12,000 | PPO @12,000 |
|---|---|---|---|---|
| Seed 1 | 4.89 m | 0.93 m | 7.08 m | 1.11 m |
| Seed 2 | 3.20 m | 2.42 m | 4.61 m | 1.75 m |
| Seed 3 | 4.39 m | 1.95 m | 5.40 m | 4.22 m |

What we learned
- For a small gait, CMA-ES wins on both samples and CPU. It reached each distance with fewer trials, and each trial is cheap: just physics. Every PPO step also runs the network, and every batch of 2,048 steps is followed by gradient updates.
- PPO's learning curve is not monotonic. In seed 3, a policy that walked 4.2 m scored 0.25 m less than 300 trials later. CMA-ES's curve only goes up because we keep the best gait; that's a real advantage when you just want a gait you can use.
- PPO found a loophole in our score. In the video, the PPO robot tips its body up on end and lunges forward instead of walking flat. Our "upright" check only asks whether the torso's top faces at least a little upward, so a body standing on end still counts. All three final PPO policies moved tipped steeply: on average 56°, 50° and 35° from level after the first second, against 13–25° for the CMA-ES gaits. Reinforcement learning optimizes exactly what you measure; reward design matters more than the algorithm. CMA-ES's fixed rhythm didn't find this trick in any of our runs.
- This is not "evolution beats RL" in general. Our CMA-ES searched a tiny, hand-picked space: a rhythm. That built-in assumption is probably a big part of why it's fast. OpenAI showed evolution strategies can train full neural-network walkers when given many CPUs [6], and Mania et al. showed simple random search of linear policies matches RL on MuJoCo locomotion benchmarks, while noting large seed-to-seed variability [7]. RL earns its cost when the controller must react: uneven ground, turning on command, getting back up.
Does the controller survive a different world?
We took the controllers saved at the end of the full runs (12,000 trials) and tested them, unchanged, in worlds they never trained in. Numbers are metres in 5 seconds for seeds 1 / 2 / 3.
| Test world | CMA-ES | PPO |
|---|---|---|
| Training world | 7.1 / 4.6 / 5.4 | 1.1 / 1.7 / 4.2 |
| Slippery floor (friction 0.5) | 4.8 / -0.5 / 3.6 | 0.7 / 3.0 / 3.7 |
| Grippy floor (friction 1.5) | 5.6 / 3.9 / 0.9 | 4.2 / 1.4 / 3.9 |
| Torso 50% heavier | 5.0 / 3.0 / 3.6 | 1.6 / -0.2 / 3.6 |
| Sideways shove (8 N for 0.1 s) | 6.4 / 4.2 / 3.5 | 1.2 / 1.2 / 4.1 |
Slippery and grippy change the friction of every surface. The shove pushes the torso sideways at t = 2.0 s. Negative = it ended up behind the start, or upside down (−1 m penalty).
Neither method came out clearly more robust. The blind CMA-ES gaits kept walking in most tests but usually lost a good part of their distance (up to about a third), and twice a gait failed outright: seed 2 scored −0.5 m on the slippery floor, and seed 3 managed only 0.9 m on the grippy one. PPO's best policy (seed 3) stayed between 3.6 and 4.2 m in every test, which is what you'd hope for from a controller that reads its sensors, but it still used the tipped-up posture, and the two weaker PPO policies were unreliable everywhere. Three seeds and four tests make this a hint, not a verdict.
Which one should you use?
| Your situation | Start with |
|---|---|
| Find a working gait fast on a laptop | CMA-ES on a small parameterized gait (sine waves, splines) |
| Tune a few dozen numbers of an existing controller (PID gains, gait timing) | CMA-ES |
| Robot must react to sensors: slopes, pushes, steering | PPO or another RL method, with a carefully tested reward |
| You can only score whole trials, not individual steps | CMA-ES (it never needs per-step rewards) |
| Many CPUs, big neural network | Either: RL, or a scalable evolution strategy [6] |
| Want both | Evolve a rhythm with CMA-ES, then train a small RL policy to correct it from sensors |
The code
Three files, about 130 lines in total. walker_env.py is shared, so both methods really see the same robot and the same score.
walker_env.py: the robot and the score
# walker_env.py - the SAME 4-legged MuJoCo robot for both CMA-ES and PPO
# pip install mujoco cma numpy gymnasium stable-baselines3
import numpy as np, mujoco, gymnasium as gym
LEG = """<body pos="{x} {y} 0"><joint name="hip{i}" type="hinge" axis="0 1 0" range="-60 60"/>
<geom type="capsule" fromto="0 0 0 0 0 -0.12" size="0.02"/>
<body pos="0 0 -0.12"><joint name="knee{i}" type="hinge" axis="0 1 0" range="-70 70"/>
<geom type="capsule" fromto="0 0 0 0 0 -0.12" size="0.018"/></body></body>"""
CORNERS = [(0.15, 0.1), (0.15, -0.1), (-0.15, 0.1), (-0.15, -0.1)]
XML = f"""<mujoco><option timestep="0.004"/>
<default><joint damping="0.5" armature="0.01"/><geom friction="1 0.005 0.0001" density="1000"/>
<position kp="8" ctrlrange="-1.2 1.2" ctrllimited="true"/></default>
<worldbody><light pos="0 0 3" dir="0 0 -1"/><geom type="plane" size="20 20 0.1" rgba=".5 .5 .5 1"/>
<body name="torso" pos="0 0 0.3"><freejoint/><geom type="box" size="0.18 0.12 0.03" density="300"/>
{''.join(LEG.format(x=x, y=y, i=i) for i, (x, y) in enumerate(CORNERS))}
</body></worldbody>
<actuator>{''.join(f'<position joint="hip{i}"/><position joint="knee{i}"/>' for i in range(4))}</actuator>
</mujoco>"""
MODEL = mujoco.MjModel.from_xml_string(XML)
N = MODEL.nu # 8 motors
SECONDS = 5.0 # every trial lasts 5 simulated seconds
STEPS = int(round(SECONDS / MODEL.opt.timestep)) # 1250 physics steps per trial
def score(d):
"""Shared fitness: metres walked forward (+x), minus 1 m if the robot ends upside down."""
upright = d.xmat[1][8] > 0
return d.qpos[0] if upright else d.qpos[0] - 1.0
class WalkerEnv(gym.Env):
"""Gymnasium wrapper for PPO. Policy acts every 5 physics steps (50 Hz), 250 actions per trial.
Reward per action = forward progress, so the episode return equals score() exactly."""
FRAME_SKIP = 5
def __init__(self):
self.model = MODEL
self.data = mujoco.MjData(MODEL)
self.action_space = gym.spaces.Box(-1.0, 1.0, (N,), np.float32)
self.observation_space = gym.spaces.Box(-np.inf, np.inf, (2 * N + 8,), np.float64)
def _obs(self):
d = self.data
# joint angles, joint speeds, torso orientation (quat), torso velocity, torso height
return np.concatenate([d.qpos[7:], d.qvel[6:], d.qpos[3:7], d.qvel[0:3], d.qpos[2:3]])
def reset(self, seed=None, options=None):
super().reset(seed=seed)
mujoco.mj_resetData(self.model, self.data)
self.t = 0
return self._obs(), {}
def step(self, action):
d = self.data
x0 = d.qpos[0]
d.ctrl[:] = 1.2 * np.clip(action, -1, 1)
for _ in range(self.FRAME_SKIP):
mujoco.mj_step(self.model, d)
self.t += 1
reward = d.qpos[0] - x0
truncated = self.t * self.FRAME_SKIP >= STEPS
if truncated and d.xmat[1][8] <= 0:
reward -= 1.0
return self._obs(), reward, False, truncated, {}
cmaes_walker.py: CMA-ES
# cmaes_walker.py - CMA-ES evolves an open-loop sine gait (25 numbers)
import sys, time, json, numpy as np, mujoco, cma
from walker_env import MODEL, N, STEPS, score
def run(genes):
amp, phase, offset = genes[:N], genes[N:2*N], genes[2*N:3*N]
freq = np.clip(1.5 + genes[-1], 0.5, 3.0)
d = mujoco.MjData(MODEL)
for _ in range(STEPS):
d.ctrl[:] = amp * np.sin(2 * np.pi * freq * d.time + phase) + offset
mujoco.mj_step(MODEL, d)
return score(d)
if __name__ == "__main__":
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
gens = int(sys.argv[2]) if len(sys.argv) > 2 else 100
es = cma.CMAEvolutionStrategy(np.zeros(3 * N + 1), 0.5, {"popsize": 24, "seed": seed, "verbose": -9})
t0, log = time.process_time(), []
for gen in range(1, gens + 1):
pop = es.ask()
dist = [run(g) for g in pop]
es.tell(pop, [-x for x in dist])
champ = run(es.result.xbest) # deterministic sim: re-run = same number
steps = gen * len(pop) * STEPS
log.append(dict(physics_steps=steps, trials=gen * len(pop), seconds=round(time.process_time() - t0, 1), distance=champ))
if gen % 10 == 0:
print(f"gen {gen:3d} trials {gen*len(pop):5d} champion {champ:5.2f} m cpu {time.process_time()-t0:6.1f} s", flush=True)
np.save(f"cmaes_seed{seed}.npy", es.result.xbest)
json.dump(log, open(f"cmaes_seed{seed}.json", "w"))
ppo_walker.py: PPO
# ppo_walker.py - PPO (stable-baselines3) learns a closed-loop neural-network policy
import sys, time, json, numpy as np, torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.callbacks import BaseCallback
from walker_env import WalkerEnv, STEPS, score
torch.set_num_threads(1)
def evaluate(model, venv):
"""Run the deterministic policy for one 5-second trial and return score()."""
env = WalkerEnv(); obs, _ = env.reset()
for _ in range(STEPS // env.FRAME_SKIP):
a, _ = model.predict(venv.normalize_obs(obs), deterministic=True)
obs, *_ = env.step(a)
return score(env.data)
class Log(BaseCallback):
def __init__(self, every, t0):
super().__init__(); self.every, self.t0, self.rows, self.next = every, t0, [], every
def _on_step(self):
if self.num_timesteps >= self.next:
self.next += self.every
dist = evaluate(self.model, self.training_env)
steps = self.num_timesteps * WalkerEnv.FRAME_SKIP
self.rows.append(dict(physics_steps=steps, trials=steps // STEPS,
seconds=round(time.process_time() - self.t0, 1), distance=dist))
print(f"physics steps {steps:8d} trials {steps//STEPS:5d} policy {dist:5.2f} m cpu {time.process_time()-self.t0:6.1f} s", flush=True)
return True
if __name__ == "__main__":
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
agent_steps = int(sys.argv[2]) if len(sys.argv) > 2 else 600_000 # = 3.0 M physics steps
venv = VecNormalize(make_vec_env(WalkerEnv, n_envs=4, seed=seed), norm_obs=True, norm_reward=True)
model = PPO("MlpPolicy", venv, n_steps=512, batch_size=64, seed=seed, verbose=0)
t0 = time.process_time(); cb = Log(24_000, t0) # log every 96 trials
model.learn(agent_steps, callback=cb)
model.save(f"ppo_seed{seed}"); venv.save(f"ppo_seed{seed}_norm.pkl")
json.dump(cb.rows, open(f"ppo_seed{seed}.json", "w"))
Run it
We ran these exact files in a fresh virtual environment (Python 3.13, mujoco 3.16.0, cma 4.5.0, numpy 2.5.4, gymnasium 1.4.0, stable-baselines3 2.9.0, torch 2.14.1, CPU only). This is the real output:
$ python -m venv venv && source venv/bin/activate $ pip install mujoco cma numpy gymnasium stable-baselines3 $ python cmaes_walker.py gen 10 trials 240 champion 1.42 m cpu 6.5 s gen 20 trials 480 champion 2.59 m cpu 13.0 s gen 30 trials 720 champion 2.59 m cpu 18.8 s gen 40 trials 960 champion 3.63 m cpu 24.8 s gen 50 trials 1200 champion 4.32 m cpu 30.8 s gen 60 trials 1440 champion 4.32 m cpu 37.0 s gen 70 trials 1680 champion 4.89 m cpu 43.0 s gen 80 trials 1920 champion 4.89 m cpu 48.7 s gen 90 trials 2160 champion 4.89 m cpu 54.9 s gen 100 trials 2400 champion 4.89 m cpu 60.6 s
$ python ppo_walker.py physics steps 120000 trials 96 policy 0.21 m cpu 14.3 s physics steps 240000 trials 192 policy -0.81 m cpu 29.9 s physics steps 360000 trials 288 policy 0.25 m cpu 45.9 s physics steps 480000 trials 384 policy 0.17 m cpu 60.5 s physics steps 600000 trials 480 policy 0.16 m cpu 75.9 s physics steps 720000 trials 576 policy 0.20 m cpu 91.9 s physics steps 840000 trials 672 policy 0.22 m cpu 107.2 s physics steps 960000 trials 768 policy -0.72 m cpu 121.9 s physics steps 1080000 trials 864 policy 0.42 m cpu 136.8 s physics steps 1200000 trials 960 policy -0.41 m cpu 152.0 s physics steps 1320000 trials 1056 policy -0.44 m cpu 166.3 s physics steps 1440000 trials 1152 policy -0.41 m cpu 181.5 s physics steps 1560000 trials 1248 policy 0.19 m cpu 196.7 s physics steps 1680000 trials 1344 policy 0.37 m cpu 211.8 s physics steps 1800000 trials 1440 policy 0.32 m cpu 226.5 s physics steps 1920000 trials 1536 policy -0.39 m cpu 241.4 s physics steps 2040000 trials 1632 policy -0.36 m cpu 256.1 s physics steps 2160000 trials 1728 policy 0.40 m cpu 270.8 s physics steps 2280000 trials 1824 policy 0.53 m cpu 285.6 s physics steps 2400000 trials 1920 policy 0.58 m cpu 300.3 s physics steps 2520000 trials 2016 policy 0.21 m cpu 315.4 s physics steps 2640000 trials 2112 policy -0.17 m cpu 329.8 s physics steps 2760000 trials 2208 policy 0.46 m cpu 344.5 s physics steps 2880000 trials 2304 policy -0.03 m cpu 359.5 s physics steps 3000000 trials 2400 policy 0.93 m cpu 374.2 s
cma also prints a one-line warning that matplotlib isn't installed; it's harmless and we left it out above. Both scripts take a seed and a length: python cmaes_walker.py 2 500 or python ppo_walker.py 2 3000000. The physics is deterministic, so the same seed and versions give the same numbers; the CPU seconds depend on your machine. Pinned versions: requirements.txt.
Evolve 2, 4, 6 and 8 legs, stairs and ice
This page uses the minimal robot. Robot Evolution Lab is the full CMA-ES code behind our videos: every leg count, stairs and ice terrain, body-shape evolution, a video recorder and pre-evolved champions.
See Robot Evolution LabGet the next robot experiment
About once a week we run an experiment like this one and publish the code. One email when it's out. Unsubscribe anytime.
Limits of this test
- One small robot, flat ground, 5-second trials, 3 seeds per method. Seeds vary a lot (see the per-seed table); treat the averages as a rough picture.
- PPO used Stable-Baselines3 defaults with normalized inputs and rewards. Tuned hyperparameters, a reward with energy and posture terms, or a different RL algorithm (SAC, TD3) would change its numbers.
- The two methods search different things: CMA-ES a 25-number rhythm, PPO a reactive network. That's the realistic choice you face, but it means the comparison is of approaches, not of optimizers on identical parameters.
- CPU time is process time on one core (cloud Linux machine, October 2026), including our periodic evaluation runs for both methods.
- Simulation only. Real motors add delay, backlash and noise; the gap between simulation and a real robot is its own problem.
FAQ
Is CMA-ES better than reinforcement learning for walking robots?
For a small, fixed gait, in our test, yes: with the same 2,400 five-second trials CMA-ES reached 4.2 m on average and PPO 1.8 m, and CMA-ES used about 6 times less CPU time. But CMA-ES only tuned 25 numbers of a blind rhythm. PPO learns a policy that reads sensors, which is what you need for rough ground, turning or recovering from pushes.
Is CMA-ES reinforcement learning?
Not in the usual sense. CMA-ES is a black-box optimizer: it only sees one score per trial and never uses the step-by-step rewards, states or gradients. It can still solve reinforcement learning problems; OpenAI's 2017 evolution strategies paper and the 2018 random search paper both trained MuJoCo walkers with black-box search instead of policy gradients.
Can CMA-ES train a neural network?
Yes, if it is small. CMA-ES keeps an n × n covariance matrix over all n parameters, so memory and work per generation grow with the square of the parameter count. Our PPO network has 12,113 parameters (6,288 in the part that picks actions); a full matrix for the action part alone would have about 40 million entries. Simpler evolution strategies (OpenAI-ES, ARS) drop the full matrix to scale further.
Why was PPO's result so jumpy?
We scored the deterministic policy in a single 5-second trial every 96 trials. Each PPO update changes the network, and a small change can turn a run into a fall, so the curve jumps. CMA-ES keeps the best gait it has found, so its curve can only go up. Averaging several evaluation episodes or saving the best checkpoint smooths PPO's curve, but the controller you actually ship is still whatever the last update produced.
Do I need a GPU for this?
No. Everything on this page ran on CPU. The PPO network is tiny; a GPU would not make it much faster because most of the time is spent stepping the physics.
Sources
- E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
- N. Hansen and A. Ostermeier, Completely Derandomized Self-Adaptation in Evolution Strategies, Evolutionary Computation 9(2), 2001.
- N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
- A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
- T. Salimans, J. Ho, X. Chen, S. Sidor, I. Sutskever, Evolution Strategies as a Scalable Alternative to Reinforcement Learning, arXiv:1703.03864, 2017.
- H. Mania, A. Guy, B. Recht, Simple random search provides a competitive approach to reinforcement learning, NeurIPS 2018, arXiv:1803.07055.
- Experiment, code and measurements: TinyBot Forge, October 2026. Raw logs available on request.
More: make a robot teach itself to walk in 45 lines · all guides