Reward hacking in reinforcement learning: fixing our PPO walking robot
In our CMA-ES vs PPO test, the PPO robot didn't really walk. It tipped its body up on end and lunged, because our score let it. So we fixed the score and ran everything again: 12 new training runs of 12,000 trials each. The first fix closed that loophole and opened a new one: both methods learned to sprint and then dive. The second fix worked. Under it, PPO went from last place to first: 4.0 m vs 2.8 m for CMA-ES, from 20 random starts per controller. It also tipped more often.
PPO seed 3 after 12,000 trials, trained under rule 2 (left) and rule 3 (right), replayed in real time. Left: it runs, then dives forward and the trial ends at 3.5 s. Right: it stays level for all 5 s and covers 7.82 m. Physics simulation (MuJoCo), not video footage.
The loophole in our first score
The robot is a 36 cm box with four two-joint legs, simulated in MuJoCo [1]. A trial is 5 simulated seconds. Our first score was metres walked forward, minus 1 m if the robot ends upside down. "Upside down" meant the torso's top faced down at the end, so a body standing almost on end still counted as upright. PPO [2] found that: its three final policies moved 56°, 50° and 35° from level on average. This is reward hacking, or specification gaming: doing what the score says rather than what we meant [4][5].
The CMA-ES gaits weren't clean either. Replayed with a 30° check, all three of our first CMA-ES champions crossed it at some point (at 4.1 s, 0.5 s, 1.9 s), even though their average tilt was only 13–25°.
Three rules, same robot
| Rule | How a 5-second trial is scored | What the robots learned |
|---|---|---|
| 1. Original | Metres forward; −1 m only if the top faces down at the end | PPO: walk tipped up on end (35–56° average) |
| 2. Stop at 30° | The trial ends the moment the torso tilts past 30°; score = metres walked so far | Sprint, then dive: 3 of 3 PPO and 2 of 3 CMA-ES controllers ended by tipping |
| 3. Tipping forfeits | Same stop, but a trial that tips scores −1 m, whatever it covered | Mostly level walking; tipping still happens, but less |
Rule 2 is the usual first fix, close to what Gymnasium's Ant does: there the episode ends when the torso leaves a healthy height range, and the agent also earns a reward for every step it stays healthy [6]. We left out the survival bonus so the score stays in metres and CMA-ES and PPO optimize the same number. Without it, ending early is free: a dive that gains 1 m in the last half second is pure profit. Rule 3 makes tipping the worst thing that can happen. Here is the code that changed:
def final_score(x, tipped_early, rule):
if rule == "forfeit" and tipped_early:
return -1.0
return x
terminated = tipped(d)
if terminated and self.rule == "forfeit":
reward += -1.0 - d.qpos[0] # return so far becomes -1 m
For PPO the last reward cancels everything walked, so an episode's total reward always equals the rule-3 score. Everything else is unchanged from the first test: same robot, CMA-ES [3] with 24 trials per generation for 500 generations, PPO with Stable-Baselines3 defaults [7] for 3 million policy steps (15 million physics steps), seeds 1–3.
Results
A controller trained under rule 2 can score well under rule 2 and still be a faller, so we judged every final controller the same strict way: rule 3, replayed from the normal start and from 20 starts with every joint nudged by up to ±0.05 rad (about 3°).
| Trained under | Method | Strict score, 20 random starts (mean of 3 seeds) | Runs that tipped | Ended the normal trial by tipping |
|---|---|---|---|---|
| Rule 1 | CMA-ES | -0.54 m | 56 / 60 | 3 of 3 |
| Rule 1 | PPO | Models not kept. Average tilt was over 30° for all three, so all three tipped: −1 each | ||
| Rule 2 | CMA-ES | -0.91 m | 59 / 60 | 2 of 3 |
| Rule 2 | PPO | 0.34 m | 52 / 60 | 3 of 3 |
| Rule 3 | CMA-ES | 2.81 m | 15 / 60 | 0 of 3 |
| Rule 3 | PPO | 4.02 m | 23 / 60 | 2 of 3 |
A tipped run scores −1, so a mean near −1 means almost every run tipped. Final controllers after 12,000 trials.

Per seed, normal start, strict score:
| Rule 2 CMA-ES | Rule 2 PPO | Rule 3 CMA-ES | Rule 3 PPO | |
|---|---|---|---|---|
| Seed 1 | 5.45 m | −1 (tipped) | 4.42 m | −1 (tipped) |
| Seed 2 | −1 (tipped) | −1 (tipped) | 4.64 m | −1 (tipped) |
| Seed 3 | −1 (tipped) | −1 (tipped) | 6.05 m | 7.82 m |

What we learned
- The first fix created the next hack. Stopping the trial at 30° removed the stand-on-end trick, but made ending early free. Both methods found the dive, CMA-ES too: it's not something only reinforcement learning does. Any optimizer pushes on whatever the score leaves open.
- Breaking the rule has to cost more than it can win. With rule 3, a dive turns a 6 m run into −1 m. That one line is what made the controllers keep the body level.
- With a fair score, PPO walked farther. In the first test PPO averaged 2.4 m vs CMA-ES 5.7 m, much of it tipped. Under rule 3 the order flipped: 4.0 m vs 2.8 m from random starts, and PPO's best policy (seed 3) covered 7.82 m level in 5 s, more than any CMA-ES gait (6.05 m).
- But PPO was less reliable. Two of its three final policies tipped from the normal start, and 92 of 300 checkpoint checks after 2,400 trials scored −1. Its policy changes with every update; CMA-ES keeps the best gait it has found. If you ship a PPO policy, keep the best checkpoint and test it from many starts.
- Open-loop gaits are fragile too. CMA-ES's rule-3 gaits never tipped from the normal start, but 15 of 60 runs tipped when every joint started up to 3° off. A blind rhythm can't correct itself.
- Watch the robot, and measure what you care about. We only caught rule 1's loophole by looking at the video, and rule 2's by measuring tilt. A score going up is not proof the robot is doing the right thing.
Ways to stop reward hacking, compared
| Fix | What it does | Risk | Tested here |
|---|---|---|---|
| End the episode when unsafe | Stops counting once the robot tilts too far | If ending is free, the agent ends on purpose (our dive) | Yes, rule 2 |
| End + make ending cost | A failed trial scores worse than any success | Harsh: a near miss scores the same as a crash, so learning is jumpier | Yes, rule 3 |
| Survival bonus | Reward for each step in a healthy state (Gymnasium Ant [6]) | Can teach standing still if the bonus outweighs progress | No |
| Tilt or energy penalty | Subtract a cost for tilt or effort at every step | One more weight to tune; too small gets gamed, too big stops motion | No |
| Watch and measure | Replay policies, track the behaviour you care about | Only finds hacks you thought to measure | Yes, how we found both |
Run it yourself
The download has the exact code we ran, plus the analysis and video scripts. From a fresh virtual environment (Python 3.13.18, mujoco 3.16.0, stable-baselines3 2.9.0, torch 2.14.1, cma 4.5.0):
$ python -m venv venv && source venv/bin/activate $ pip install -r requirements.txt $ python cmaes_walker.py gen 10 trials 240 champion 1.01 m cpu 2.2 s gen 20 trials 480 champion 1.86 m cpu 5.0 s gen 30 trials 720 champion 1.86 m cpu 8.0 s gen 40 trials 960 champion 1.86 m cpu 11.8 s gen 50 trials 1200 champion 2.04 m cpu 16.0 s gen 60 trials 1440 champion 3.16 m cpu 20.1 s gen 70 trials 1680 champion 3.16 m cpu 24.2 s gen 80 trials 1920 champion 3.16 m cpu 28.3 s gen 90 trials 2160 champion 3.16 m cpu 32.3 s gen 100 trials 2400 champion 3.16 m cpu 36.1 s
$ python ppo_walker.py physics steps 120000 trials 96 policy -0.06 m cpu 12.7 s physics steps 240000 trials 192 policy 0.17 m cpu 26.2 s physics steps 360000 trials 288 policy 0.59 m cpu 39.4 s physics steps 480000 trials 384 policy 0.12 m cpu 52.0 s physics steps 600000 trials 480 policy 3.31 m cpu 65.2 s physics steps 720000 trials 576 policy 4.57 m cpu 78.3 s physics steps 840000 trials 672 policy 4.69 m cpu 91.8 s physics steps 960000 trials 768 policy 4.62 m cpu 104.7 s physics steps 1080000 trials 864 policy 5.00 m cpu 118.3 s physics steps 1200000 trials 960 policy 5.44 m cpu 132.0 s physics steps 1320000 trials 1056 policy -1.00 m cpu 144.9 s physics steps 1440000 trials 1152 policy 6.57 m cpu 158.7 s physics steps 1560000 trials 1248 policy 6.45 m cpu 172.5 s physics steps 1680000 trials 1344 policy 6.29 m cpu 186.2 s physics steps 1800000 trials 1440 policy 6.07 m cpu 199.7 s physics steps 1920000 trials 1536 policy 6.16 m cpu 213.4 s physics steps 2040000 trials 1632 policy 6.44 m cpu 227.6 s physics steps 2160000 trials 1728 policy 6.42 m cpu 240.6 s physics steps 2280000 trials 1824 policy 7.08 m cpu 254.3 s physics steps 2400000 trials 1920 policy 7.04 m cpu 267.4 s physics steps 2520000 trials 2016 policy 7.37 m cpu 280.9 s physics steps 2640000 trials 2112 policy 7.24 m cpu 293.9 s physics steps 2760000 trials 2208 policy -1.00 m cpu 307.5 s physics steps 2880000 trials 2304 policy 7.22 m cpu 320.6 s physics steps 3000000 trials 2400 policy -1.00 m cpu 333.2 s
That is real output, with cma's one-line "matplotlib not installed" warning removed. The defaults are rule 3, seed 1, 2,400 trials; these lines match the first 2,400 trials of our full runs exactly, because the physics is deterministic. Note PPO's last line: −1, a tipped trial. Add a seed, length and rule: python ppo_walker.py 3 3000000 forfeit. CPU seconds depend on your machine. No GPU needed.
Download reward-hacking-walker.zip or the single files: walker_env.py, cmaes_walker.py, ppo_walker.py, requirements.txt
Next: can random worlds stop the falls?
We retrained both methods on random floor friction and start poses. It cut CMA-ES tips by two thirds and did little for PPO.
See the follow-upLimits
- Three seeds per method and rule; seeds vary a lot (see the per-seed table). Treat the averages as a rough picture.
- PPO used Stable-Baselines3 defaults. Tuned settings, a survival bonus or tilt penalty, or keeping the best checkpoint would change its numbers.
- 30° is our choice, not a standard. CPU time is process time on one core of a cloud Linux machine, November 2026.
- Simulation only. A real robot adds motor delay, backlash and sensor noise.
Evolve your own walkers
Robot Evolution Lab is the full CMA-ES code behind our videos: 2 to 8 legs, stairs and ice, body-shape evolution and a video recorder.
See Robot Evolution LabGet the next robot experiment
One email when we publish a new experiment, about once a week. Unsubscribe anytime.
FAQ
What is reward hacking in reinforcement learning?
The agent finds a way to get a high score that the designer didn't intend, because the reward only approximately describes the goal. DeepMind calls the broader idea specification gaming: behaviour that satisfies the literal objective without achieving the intended outcome.
How did our PPO walking robot hack its reward?
The original score gave metres walked forward and only took 1 m off if the robot ended upside down. PPO learned to tip its body up on end and lunge forward: its three final policies moved 56°, 50° and 35° from level on average.
Does ending the episode when the robot falls fix reward hacking?
Not on its own, in our test. When we stopped the trial at 30° of tilt and kept the distance covered so far, both methods learned to sprint and then dive: 3 of 3 PPO policies and 2 of 3 CMA-ES gaits ended their trial by tipping. Only when tipping cost the whole run (score −1 m) did they keep the body level.
Is PPO better than CMA-ES once the reward is fixed?
With the forfeit rule and 20 slightly randomized starts per controller, PPO averaged 4.0 m and CMA-ES 2.8 m (3 seeds each). PPO was faster but less consistent: 23 of its 60 runs tipped versus 15 of 60 for CMA-ES, and 92 of its 300 checkpoint evaluations after 2,400 trials scored −1.
Why not just add a tilt penalty to the reward?
You can, and many environments do (posture, energy and survival terms). We tested only rules that keep the score in metres so CMA-ES and PPO optimize the same number. A penalty weight is one more knob that can be gamed if it's too small or can stop the robot moving if it's too big.
Sources
- E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
- N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
- V. Krakovna et al., Specification gaming: the flip side of AI ingenuity, DeepMind, April 21, 2020.
- D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané, Concrete Problems in AI Safety, arXiv:1606.06565, 2016 (section on reward hacking).
- Farama Foundation, Gymnasium Ant environment (healthy_reward, terminate_when_unhealthy, healthy_z_range).
- A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
- Experiment, code and measurements: TinyBot Forge, November 2026. Every number on this page comes from the code in the download.
More: CMA-ES vs PPO, the first test · make a robot teach itself to walk · all guides