TINYBOT FORGE
Home › Learn › CMA-ES vs reinforcement learning

CMA-ES vs reinforcement learning (PPO) for a walking robot: we tested both

Our experimentMuJoCo + Python3 seeds eachUpdated October 19, 2026

Should you evolve a walking gait with CMA-ES or train it with reinforcement learning? We gave both the same four-legged robot, the same 5-second test and the same number of tries. With 2,400 trials, CMA-ES walked 4.2 m on average and PPO 1.8 m, and CMA-ES needed about 6× less CPU time. Then the rest of the story got more interesting.

Seed 1 of each method after 12,000 trials, replayed in real time. Physics simulation (MuJoCo), not video footage. The PPO robot learned to tip its body up on end and lunge forward; more on that below.

4.2 m vs 1.8 mDistance in 5 s after 2,400 trials, CMA-ES vs PPO (mean of 3 seeds)
63 s vs 376 sCPU time for those 2,400 trials, one core
25 vs 6,288Numbers being tuned: sine-gait genes vs PPO's action network

The setup: one robot, one test, two brains

The robot is the same one from our guide make a robot teach itself to walk: a 36 cm box torso, four legs, a hip and a knee motor on each leg, simulated in MuJoCo. Every trial lasts 5 simulated seconds (1,250 physics steps). The score is the distance walked forward, minus 1 m if the robot ends upside down. Both methods are optimizing exactly this number.

CMA-ESPPO (reinforcement learning)
ControllerEach motor follows a sine wave. 8 amplitudes + 8 phases + 8 offsets + 1 frequency = 25 numbersNeural network (2 hidden layers of 64) maps 24 sensor values to 8 motor targets, 50 times a second
Sees sensors?No, it plays the same rhythm blind (open loop)Yes: joint angles and speeds, torso tilt, speed and height (closed loop)
Learns fromOne number per trial: the final scoreThe reward at every step (forward progress), plus a learned value estimate
AlgorithmCMA-ES [2][3], the cma package, 24 trials per generationPPO [4], Stable-Baselines3 [5] default settings, 4 parallel environments of 512 steps (2,048 per update), normalized observations and rewards
BudgetSame physics steps for both. The main comparison is at 3 million steps (2,400 trials, CMA-ES's 100-generation default); we kept both running to 15 million (12,000 trials). Seeds 1, 2 and 3.

This is a deliberately practical comparison: each method as a hobbyist would use it, with its usual controller and default settings, on an ordinary CPU. It is not a tuned benchmark.

Update: we fixed the loophole

We fixed this score twice and re-ran both methods. The first fix bred a new hack; with the second, PPO walked farther than CMA-ES.

See the follow-up

Results

Line chart of distance walked vs physics steps up to 3 million. CMA-ES rises steadily to about 4.2 m; PPO climbs slower and jumps up and down, ending near 1.8 m.
Thin lines: the 3 seeds. Thick line: their mean. CMA-ES is scored on the best gait found so far; PPO on its current policy, one deterministic trial every 96 trials.
Same robot, same testCMA-ESPPO
Distance after 2,400 trials (mean of 3)4.16 m1.77 m
CPU time for 2,400 trials63 s376 s
Trials until it first walks 1 m (median)1442,016
Trials until 2 m4802,496
Trials until 3 m9603,168
Trials until 4 m1,7284,800
Distance after 12,000 trials (mean)5.69 m2.36 m
Best single evaluation, any time (mean)5.69 m4.59 m

CMA-ES is logged every generation (24 trials), PPO every 96 trials, so PPO's "trials until" numbers are rounded up to the next 96. "Distance after 12,000 trials" is the controller saved at the end of the run (for PPO that is after the last update, so it can differ from the curve's last point).

Per seedCMA-ES @2,400PPO @2,400CMA-ES @12,000PPO @12,000
Seed 14.89 m0.93 m7.08 m1.11 m
Seed 23.20 m2.42 m4.61 m1.75 m
Seed 34.39 m1.95 m5.40 m4.22 m
Line chart to 15 million physics steps: CMA-ES keeps improving slowly; PPO swings between near zero and over 4 m.
The full runs. The dashed line marks the 2,400-trial point used above.

What we learned

Does the controller survive a different world?

We took the controllers saved at the end of the full runs (12,000 trials) and tested them, unchanged, in worlds they never trained in. Numbers are metres in 5 seconds for seeds 1 / 2 / 3.

Test worldCMA-ESPPO
Training world7.1 / 4.6 / 5.41.1 / 1.7 / 4.2
Slippery floor (friction 0.5)4.8 / -0.5 / 3.60.7 / 3.0 / 3.7
Grippy floor (friction 1.5)5.6 / 3.9 / 0.94.2 / 1.4 / 3.9
Torso 50% heavier5.0 / 3.0 / 3.61.6 / -0.2 / 3.6
Sideways shove (8 N for 0.1 s)6.4 / 4.2 / 3.51.2 / 1.2 / 4.1

Slippery and grippy change the friction of every surface. The shove pushes the torso sideways at t = 2.0 s. Negative = it ended up behind the start, or upside down (−1 m penalty).

Neither method came out clearly more robust. The blind CMA-ES gaits kept walking in most tests but usually lost a good part of their distance (up to about a third), and twice a gait failed outright: seed 2 scored −0.5 m on the slippery floor, and seed 3 managed only 0.9 m on the grippy one. PPO's best policy (seed 3) stayed between 3.6 and 4.2 m in every test, which is what you'd hope for from a controller that reads its sensors, but it still used the tipped-up posture, and the two weaker PPO policies were unreliable everywhere. Three seeds and four tests make this a hint, not a verdict.

Which one should you use?

Your situationStart with
Find a working gait fast on a laptopCMA-ES on a small parameterized gait (sine waves, splines)
Tune a few dozen numbers of an existing controller (PID gains, gait timing)CMA-ES
Robot must react to sensors: slopes, pushes, steeringPPO or another RL method, with a carefully tested reward
You can only score whole trials, not individual stepsCMA-ES (it never needs per-step rewards)
Many CPUs, big neural networkEither: RL, or a scalable evolution strategy [6]
Want bothEvolve a rhythm with CMA-ES, then train a small RL policy to correct it from sensors

The code

Three files, about 130 lines in total. walker_env.py is shared, so both methods really see the same robot and the same score.

walker_env.py: the robot and the score

# walker_env.py - the SAME 4-legged MuJoCo robot for both CMA-ES and PPO
# pip install mujoco cma numpy gymnasium stable-baselines3
import numpy as np, mujoco, gymnasium as gym

LEG = """<body pos="{x} {y} 0"><joint name="hip{i}" type="hinge" axis="0 1 0" range="-60 60"/>
  <geom type="capsule" fromto="0 0 0 0 0 -0.12" size="0.02"/>
  <body pos="0 0 -0.12"><joint name="knee{i}" type="hinge" axis="0 1 0" range="-70 70"/>
    <geom type="capsule" fromto="0 0 0 0 0 -0.12" size="0.018"/></body></body>"""
CORNERS = [(0.15, 0.1), (0.15, -0.1), (-0.15, 0.1), (-0.15, -0.1)]
XML = f"""<mujoco><option timestep="0.004"/>
<default><joint damping="0.5" armature="0.01"/><geom friction="1 0.005 0.0001" density="1000"/>
  <position kp="8" ctrlrange="-1.2 1.2" ctrllimited="true"/></default>
<worldbody><light pos="0 0 3" dir="0 0 -1"/><geom type="plane" size="20 20 0.1" rgba=".5 .5 .5 1"/>
 <body name="torso" pos="0 0 0.3"><freejoint/><geom type="box" size="0.18 0.12 0.03" density="300"/>
  {''.join(LEG.format(x=x, y=y, i=i) for i, (x, y) in enumerate(CORNERS))}
 </body></worldbody>
<actuator>{''.join(f'<position joint="hip{i}"/><position joint="knee{i}"/>' for i in range(4))}</actuator>
</mujoco>"""

MODEL = mujoco.MjModel.from_xml_string(XML)
N = MODEL.nu            # 8 motors
SECONDS = 5.0           # every trial lasts 5 simulated seconds
STEPS = int(round(SECONDS / MODEL.opt.timestep))   # 1250 physics steps per trial

def score(d):
    """Shared fitness: metres walked forward (+x), minus 1 m if the robot ends upside down."""
    upright = d.xmat[1][8] > 0
    return d.qpos[0] if upright else d.qpos[0] - 1.0

class WalkerEnv(gym.Env):
    """Gymnasium wrapper for PPO. Policy acts every 5 physics steps (50 Hz), 250 actions per trial.
    Reward per action = forward progress, so the episode return equals score() exactly."""
    FRAME_SKIP = 5
    def __init__(self):
        self.model = MODEL
        self.data = mujoco.MjData(MODEL)
        self.action_space = gym.spaces.Box(-1.0, 1.0, (N,), np.float32)
        self.observation_space = gym.spaces.Box(-np.inf, np.inf, (2 * N + 8,), np.float64)
    def _obs(self):
        d = self.data
        # joint angles, joint speeds, torso orientation (quat), torso velocity, torso height
        return np.concatenate([d.qpos[7:], d.qvel[6:], d.qpos[3:7], d.qvel[0:3], d.qpos[2:3]])
    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        mujoco.mj_resetData(self.model, self.data)
        self.t = 0
        return self._obs(), {}
    def step(self, action):
        d = self.data
        x0 = d.qpos[0]
        d.ctrl[:] = 1.2 * np.clip(action, -1, 1)
        for _ in range(self.FRAME_SKIP):
            mujoco.mj_step(self.model, d)
        self.t += 1
        reward = d.qpos[0] - x0
        truncated = self.t * self.FRAME_SKIP >= STEPS
        if truncated and d.xmat[1][8] <= 0:
            reward -= 1.0
        return self._obs(), reward, False, truncated, {}
Download walker_env.py

cmaes_walker.py: CMA-ES

# cmaes_walker.py - CMA-ES evolves an open-loop sine gait (25 numbers)
import sys, time, json, numpy as np, mujoco, cma
from walker_env import MODEL, N, STEPS, score

def run(genes):
    amp, phase, offset = genes[:N], genes[N:2*N], genes[2*N:3*N]
    freq = np.clip(1.5 + genes[-1], 0.5, 3.0)
    d = mujoco.MjData(MODEL)
    for _ in range(STEPS):
        d.ctrl[:] = amp * np.sin(2 * np.pi * freq * d.time + phase) + offset
        mujoco.mj_step(MODEL, d)
    return score(d)

if __name__ == "__main__":
    seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
    gens = int(sys.argv[2]) if len(sys.argv) > 2 else 100
    es = cma.CMAEvolutionStrategy(np.zeros(3 * N + 1), 0.5, {"popsize": 24, "seed": seed, "verbose": -9})
    t0, log = time.process_time(), []
    for gen in range(1, gens + 1):
        pop = es.ask()
        dist = [run(g) for g in pop]
        es.tell(pop, [-x for x in dist])
        champ = run(es.result.xbest)          # deterministic sim: re-run = same number
        steps = gen * len(pop) * STEPS
        log.append(dict(physics_steps=steps, trials=gen * len(pop), seconds=round(time.process_time() - t0, 1), distance=champ))
        if gen % 10 == 0:
            print(f"gen {gen:3d}  trials {gen*len(pop):5d}  champion {champ:5.2f} m  cpu {time.process_time()-t0:6.1f} s", flush=True)
    np.save(f"cmaes_seed{seed}.npy", es.result.xbest)
    json.dump(log, open(f"cmaes_seed{seed}.json", "w"))
Download cmaes_walker.py

ppo_walker.py: PPO

# ppo_walker.py - PPO (stable-baselines3) learns a closed-loop neural-network policy
import sys, time, json, numpy as np, torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.callbacks import BaseCallback
from walker_env import WalkerEnv, STEPS, score

torch.set_num_threads(1)

def evaluate(model, venv):
    """Run the deterministic policy for one 5-second trial and return score()."""
    env = WalkerEnv(); obs, _ = env.reset()
    for _ in range(STEPS // env.FRAME_SKIP):
        a, _ = model.predict(venv.normalize_obs(obs), deterministic=True)
        obs, *_ = env.step(a)
    return score(env.data)

class Log(BaseCallback):
    def __init__(self, every, t0):
        super().__init__(); self.every, self.t0, self.rows, self.next = every, t0, [], every
    def _on_step(self):
        if self.num_timesteps >= self.next:
            self.next += self.every
            dist = evaluate(self.model, self.training_env)
            steps = self.num_timesteps * WalkerEnv.FRAME_SKIP
            self.rows.append(dict(physics_steps=steps, trials=steps // STEPS,
                                  seconds=round(time.process_time() - self.t0, 1), distance=dist))
            print(f"physics steps {steps:8d}  trials {steps//STEPS:5d}  policy {dist:5.2f} m  cpu {time.process_time()-self.t0:6.1f} s", flush=True)
        return True

if __name__ == "__main__":
    seed = int(sys.argv[1]) if len(sys.argv) > 1 else 1
    agent_steps = int(sys.argv[2]) if len(sys.argv) > 2 else 600_000   # = 3.0 M physics steps
    venv = VecNormalize(make_vec_env(WalkerEnv, n_envs=4, seed=seed), norm_obs=True, norm_reward=True)
    model = PPO("MlpPolicy", venv, n_steps=512, batch_size=64, seed=seed, verbose=0)
    t0 = time.process_time(); cb = Log(24_000, t0)       # log every 96 trials
    model.learn(agent_steps, callback=cb)
    model.save(f"ppo_seed{seed}"); venv.save(f"ppo_seed{seed}_norm.pkl")
    json.dump(cb.rows, open(f"ppo_seed{seed}.json", "w"))
Download ppo_walker.py

Run it

We ran these exact files in a fresh virtual environment (Python 3.13, mujoco 3.16.0, cma 4.5.0, numpy 2.5.4, gymnasium 1.4.0, stable-baselines3 2.9.0, torch 2.14.1, CPU only). This is the real output:

$ python -m venv venv && source venv/bin/activate
$ pip install mujoco cma numpy gymnasium stable-baselines3
$ python cmaes_walker.py
gen  10  trials   240  champion  1.42 m  cpu    6.5 s
gen  20  trials   480  champion  2.59 m  cpu   13.0 s
gen  30  trials   720  champion  2.59 m  cpu   18.8 s
gen  40  trials   960  champion  3.63 m  cpu   24.8 s
gen  50  trials  1200  champion  4.32 m  cpu   30.8 s
gen  60  trials  1440  champion  4.32 m  cpu   37.0 s
gen  70  trials  1680  champion  4.89 m  cpu   43.0 s
gen  80  trials  1920  champion  4.89 m  cpu   48.7 s
gen  90  trials  2160  champion  4.89 m  cpu   54.9 s
gen 100  trials  2400  champion  4.89 m  cpu   60.6 s
$ python ppo_walker.py
physics steps   120000  trials    96  policy  0.21 m  cpu   14.3 s
physics steps   240000  trials   192  policy -0.81 m  cpu   29.9 s
physics steps   360000  trials   288  policy  0.25 m  cpu   45.9 s
physics steps   480000  trials   384  policy  0.17 m  cpu   60.5 s
physics steps   600000  trials   480  policy  0.16 m  cpu   75.9 s
physics steps   720000  trials   576  policy  0.20 m  cpu   91.9 s
physics steps   840000  trials   672  policy  0.22 m  cpu  107.2 s
physics steps   960000  trials   768  policy -0.72 m  cpu  121.9 s
physics steps  1080000  trials   864  policy  0.42 m  cpu  136.8 s
physics steps  1200000  trials   960  policy -0.41 m  cpu  152.0 s
physics steps  1320000  trials  1056  policy -0.44 m  cpu  166.3 s
physics steps  1440000  trials  1152  policy -0.41 m  cpu  181.5 s
physics steps  1560000  trials  1248  policy  0.19 m  cpu  196.7 s
physics steps  1680000  trials  1344  policy  0.37 m  cpu  211.8 s
physics steps  1800000  trials  1440  policy  0.32 m  cpu  226.5 s
physics steps  1920000  trials  1536  policy -0.39 m  cpu  241.4 s
physics steps  2040000  trials  1632  policy -0.36 m  cpu  256.1 s
physics steps  2160000  trials  1728  policy  0.40 m  cpu  270.8 s
physics steps  2280000  trials  1824  policy  0.53 m  cpu  285.6 s
physics steps  2400000  trials  1920  policy  0.58 m  cpu  300.3 s
physics steps  2520000  trials  2016  policy  0.21 m  cpu  315.4 s
physics steps  2640000  trials  2112  policy -0.17 m  cpu  329.8 s
physics steps  2760000  trials  2208  policy  0.46 m  cpu  344.5 s
physics steps  2880000  trials  2304  policy -0.03 m  cpu  359.5 s
physics steps  3000000  trials  2400  policy  0.93 m  cpu  374.2 s

cma also prints a one-line warning that matplotlib isn't installed; it's harmless and we left it out above. Both scripts take a seed and a length: python cmaes_walker.py 2 500 or python ppo_walker.py 2 3000000. The physics is deterministic, so the same seed and versions give the same numbers; the CPU seconds depend on your machine. Pinned versions: requirements.txt.

Evolve 2, 4, 6 and 8 legs, stairs and ice

This page uses the minimal robot. Robot Evolution Lab is the full CMA-ES code behind our videos: every leg count, stairs and ice terrain, body-shape evolution, a video recorder and pre-evolved champions.

See Robot Evolution Lab

Get the next robot experiment

About once a week we run an experiment like this one and publish the code. One email when it's out. Unsubscribe anytime.

Free. No spam, one email per experiment.

Limits of this test

FAQ

Is CMA-ES better than reinforcement learning for walking robots?

For a small, fixed gait, in our test, yes: with the same 2,400 five-second trials CMA-ES reached 4.2 m on average and PPO 1.8 m, and CMA-ES used about 6 times less CPU time. But CMA-ES only tuned 25 numbers of a blind rhythm. PPO learns a policy that reads sensors, which is what you need for rough ground, turning or recovering from pushes.

Is CMA-ES reinforcement learning?

Not in the usual sense. CMA-ES is a black-box optimizer: it only sees one score per trial and never uses the step-by-step rewards, states or gradients. It can still solve reinforcement learning problems; OpenAI's 2017 evolution strategies paper and the 2018 random search paper both trained MuJoCo walkers with black-box search instead of policy gradients.

Can CMA-ES train a neural network?

Yes, if it is small. CMA-ES keeps an n × n covariance matrix over all n parameters, so memory and work per generation grow with the square of the parameter count. Our PPO network has 12,113 parameters (6,288 in the part that picks actions); a full matrix for the action part alone would have about 40 million entries. Simpler evolution strategies (OpenAI-ES, ARS) drop the full matrix to scale further.

Why was PPO's result so jumpy?

We scored the deterministic policy in a single 5-second trial every 96 trials. Each PPO update changes the network, and a small change can turn a run into a fall, so the curve jumps. CMA-ES keeps the best gait it has found, so its curve can only go up. Averaging several evaluation episodes or saving the best checkpoint smooths PPO's curve, but the controller you actually ship is still whatever the last update produced.

Do I need a GPU for this?

No. Everything on this page ran on CPU. The PPO network is tiny; a GPU would not make it much faster because most of the time is spent stepping the physics.

Sources

  1. E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
  2. N. Hansen and A. Ostermeier, Completely Derandomized Self-Adaptation in Evolution Strategies, Evolutionary Computation 9(2), 2001.
  3. N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
  4. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
  5. A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
  6. T. Salimans, J. Ho, X. Chen, S. Sidor, I. Sutskever, Evolution Strategies as a Scalable Alternative to Reinforcement Learning, arXiv:1703.03864, 2017.
  7. H. Mania, A. Guy, B. Recht, Simple random search provides a competitive approach to reinforcement learning, NeurIPS 2018, arXiv:1803.07055.
  8. Experiment, code and measurements: TinyBot Forge, October 2026. Raw logs available on request.

More: make a robot teach itself to walk in 45 lines · all guides