TINYBOT FORGE
Home › Learn › Reward hacking

Reward hacking in reinforcement learning: fixing our PPO walking robot

Our experimentMuJoCo + Python3 seeds × 3 rules × 2 methodsUpdated November 9, 2026

In our CMA-ES vs PPO test, the PPO robot didn't really walk. It tipped its body up on end and lunged, because our score let it. So we fixed the score and ran everything again: 12 new training runs of 12,000 trials each. The first fix closed that loophole and opened a new one: both methods learned to sprint and then dive. The second fix worked. Under it, PPO went from last place to first: 4.0 m vs 2.8 m for CMA-ES, from 20 random starts per controller. It also tipped more often.

PPO seed 3 after 12,000 trials, trained under rule 2 (left) and rule 3 (right), replayed in real time. Left: it runs, then dives forward and the trial ends at 3.5 s. Right: it stays level for all 5 s and covers 7.82 m. Physics simulation (MuJoCo), not video footage.

52 / 60 → 23 / 60PPO runs that tipped past 30° (20 random starts × 3 seeds), rule 2 → rule 3
4.0 m vs 2.8 mStrict score under rule 3, PPO vs CMA-ES, mean of 60 runs each
92 of 300PPO checkpoint checks after 2,400 trials that still scored −1 (CMA-ES: 0)

The loophole in our first score

The robot is a 36 cm box with four two-joint legs, simulated in MuJoCo [1]. A trial is 5 simulated seconds. Our first score was metres walked forward, minus 1 m if the robot ends upside down. "Upside down" meant the torso's top faced down at the end, so a body standing almost on end still counted as upright. PPO [2] found that: its three final policies moved 56°, 50° and 35° from level on average. This is reward hacking, or specification gaming: doing what the score says rather than what we meant [4][5].

The CMA-ES gaits weren't clean either. Replayed with a 30° check, all three of our first CMA-ES champions crossed it at some point (at 4.1 s, 0.5 s, 1.9 s), even though their average tilt was only 13–25°.

Three rules, same robot

RuleHow a 5-second trial is scoredWhat the robots learned
1. OriginalMetres forward; −1 m only if the top faces down at the endPPO: walk tipped up on end (35–56° average)
2. Stop at 30°The trial ends the moment the torso tilts past 30°; score = metres walked so farSprint, then dive: 3 of 3 PPO and 2 of 3 CMA-ES controllers ended by tipping
3. Tipping forfeitsSame stop, but a trial that tips scores −1 m, whatever it coveredMostly level walking; tipping still happens, but less

Rule 2 is the usual first fix, close to what Gymnasium's Ant does: there the episode ends when the torso leaves a healthy height range, and the agent also earns a reward for every step it stays healthy [6]. We left out the survival bonus so the score stays in metres and CMA-ES and PPO optimize the same number. Without it, ending early is free: a dive that gains 1 m in the last half second is pure profit. Rule 3 makes tipping the worst thing that can happen. Here is the code that changed:

def final_score(x, tipped_early, rule):
    if rule == "forfeit" and tipped_early:
        return -1.0
    return x
        terminated = tipped(d)
        if terminated and self.rule == "forfeit":
            reward += -1.0 - d.qpos[0]      # return so far becomes -1 m

For PPO the last reward cancels everything walked, so an episode's total reward always equals the rule-3 score. Everything else is unchanged from the first test: same robot, CMA-ES [3] with 24 trials per generation for 500 generations, PPO with Stable-Baselines3 defaults [7] for 3 million policy steps (15 million physics steps), seeds 1–3.

Results

A controller trained under rule 2 can score well under rule 2 and still be a faller, so we judged every final controller the same strict way: rule 3, replayed from the normal start and from 20 starts with every joint nudged by up to ±0.05 rad (about 3°).

Trained underMethodStrict score, 20 random starts (mean of 3 seeds)Runs that tippedEnded the normal trial by tipping
Rule 1CMA-ES-0.54 m56 / 603 of 3
Rule 1PPOModels not kept. Average tilt was over 30° for all three, so all three tipped: −1 each
Rule 2CMA-ES-0.91 m59 / 602 of 3
Rule 2PPO0.34 m52 / 603 of 3
Rule 3CMA-ES2.81 m15 / 600 of 3
Rule 3PPO4.02 m23 / 602 of 3

A tipped run scores −1, so a mean near −1 means almost every run tipped. Final controllers after 12,000 trials.

Two plots of torso tilt over 5 seconds for the final controllers. Under rule 2 most lines cross the 30 degree line and some go to 90 degrees. Under rule 3 the CMA-ES lines stay between 5 and 25 degrees; one PPO line stays low and two cross 30 degrees.
Tilt of every final controller from the normal start, replayed for the full 5 s even after crossing 30°. Rule 2 ends in dives; rule 3 mostly stays level.

Per seed, normal start, strict score:

Rule 2 CMA-ESRule 2 PPORule 3 CMA-ESRule 3 PPO
Seed 15.45 m−1 (tipped)4.42 m−1 (tipped)
Seed 2−1 (tipped)−1 (tipped)4.64 m−1 (tipped)
Seed 3−1 (tipped)−1 (tipped)6.05 m7.82 m
Learning curves under rule 3. CMA-ES rises slowly and smoothly to about 5 m. PPO rises to 6 to 8 m within 2 million steps but jumps down to -1 often.
Rule 3 learning curves. CMA-ES is scored on the best gait found so far; PPO on its current policy, one deterministic trial every 96 trials. Thin lines: seeds; thick: mean.

What we learned

Ways to stop reward hacking, compared

FixWhat it doesRiskTested here
End the episode when unsafeStops counting once the robot tilts too farIf ending is free, the agent ends on purpose (our dive)Yes, rule 2
End + make ending costA failed trial scores worse than any successHarsh: a near miss scores the same as a crash, so learning is jumpierYes, rule 3
Survival bonusReward for each step in a healthy state (Gymnasium Ant [6])Can teach standing still if the bonus outweighs progressNo
Tilt or energy penaltySubtract a cost for tilt or effort at every stepOne more weight to tune; too small gets gamed, too big stops motionNo
Watch and measureReplay policies, track the behaviour you care aboutOnly finds hacks you thought to measureYes, how we found both

Run it yourself

The download has the exact code we ran, plus the analysis and video scripts. From a fresh virtual environment (Python 3.13.18, mujoco 3.16.0, stable-baselines3 2.9.0, torch 2.14.1, cma 4.5.0):

$ python -m venv venv && source venv/bin/activate
$ pip install -r requirements.txt
$ python cmaes_walker.py
gen  10  trials   240  champion  1.01 m  cpu    2.2 s
gen  20  trials   480  champion  1.86 m  cpu    5.0 s
gen  30  trials   720  champion  1.86 m  cpu    8.0 s
gen  40  trials   960  champion  1.86 m  cpu   11.8 s
gen  50  trials  1200  champion  2.04 m  cpu   16.0 s
gen  60  trials  1440  champion  3.16 m  cpu   20.1 s
gen  70  trials  1680  champion  3.16 m  cpu   24.2 s
gen  80  trials  1920  champion  3.16 m  cpu   28.3 s
gen  90  trials  2160  champion  3.16 m  cpu   32.3 s
gen 100  trials  2400  champion  3.16 m  cpu   36.1 s
$ python ppo_walker.py
physics steps   120000  trials    96  policy -0.06 m  cpu   12.7 s
physics steps   240000  trials   192  policy  0.17 m  cpu   26.2 s
physics steps   360000  trials   288  policy  0.59 m  cpu   39.4 s
physics steps   480000  trials   384  policy  0.12 m  cpu   52.0 s
physics steps   600000  trials   480  policy  3.31 m  cpu   65.2 s
physics steps   720000  trials   576  policy  4.57 m  cpu   78.3 s
physics steps   840000  trials   672  policy  4.69 m  cpu   91.8 s
physics steps   960000  trials   768  policy  4.62 m  cpu  104.7 s
physics steps  1080000  trials   864  policy  5.00 m  cpu  118.3 s
physics steps  1200000  trials   960  policy  5.44 m  cpu  132.0 s
physics steps  1320000  trials  1056  policy -1.00 m  cpu  144.9 s
physics steps  1440000  trials  1152  policy  6.57 m  cpu  158.7 s
physics steps  1560000  trials  1248  policy  6.45 m  cpu  172.5 s
physics steps  1680000  trials  1344  policy  6.29 m  cpu  186.2 s
physics steps  1800000  trials  1440  policy  6.07 m  cpu  199.7 s
physics steps  1920000  trials  1536  policy  6.16 m  cpu  213.4 s
physics steps  2040000  trials  1632  policy  6.44 m  cpu  227.6 s
physics steps  2160000  trials  1728  policy  6.42 m  cpu  240.6 s
physics steps  2280000  trials  1824  policy  7.08 m  cpu  254.3 s
physics steps  2400000  trials  1920  policy  7.04 m  cpu  267.4 s
physics steps  2520000  trials  2016  policy  7.37 m  cpu  280.9 s
physics steps  2640000  trials  2112  policy  7.24 m  cpu  293.9 s
physics steps  2760000  trials  2208  policy -1.00 m  cpu  307.5 s
physics steps  2880000  trials  2304  policy  7.22 m  cpu  320.6 s
physics steps  3000000  trials  2400  policy -1.00 m  cpu  333.2 s

That is real output, with cma's one-line "matplotlib not installed" warning removed. The defaults are rule 3, seed 1, 2,400 trials; these lines match the first 2,400 trials of our full runs exactly, because the physics is deterministic. Note PPO's last line: −1, a tipped trial. Add a seed, length and rule: python ppo_walker.py 3 3000000 forfeit. CPU seconds depend on your machine. No GPU needed.

Download reward-hacking-walker.zip or the single files: walker_env.py, cmaes_walker.py, ppo_walker.py, requirements.txt

Next: can random worlds stop the falls?

We retrained both methods on random floor friction and start poses. It cut CMA-ES tips by two thirds and did little for PPO.

See the follow-up

Limits

Evolve your own walkers

Robot Evolution Lab is the full CMA-ES code behind our videos: 2 to 8 legs, stairs and ice, body-shape evolution and a video recorder.

See Robot Evolution Lab

Get the next robot experiment

One email when we publish a new experiment, about once a week. Unsubscribe anytime.

Free. No spam, one email per experiment.

FAQ

What is reward hacking in reinforcement learning?

The agent finds a way to get a high score that the designer didn't intend, because the reward only approximately describes the goal. DeepMind calls the broader idea specification gaming: behaviour that satisfies the literal objective without achieving the intended outcome.

How did our PPO walking robot hack its reward?

The original score gave metres walked forward and only took 1 m off if the robot ended upside down. PPO learned to tip its body up on end and lunge forward: its three final policies moved 56°, 50° and 35° from level on average.

Does ending the episode when the robot falls fix reward hacking?

Not on its own, in our test. When we stopped the trial at 30° of tilt and kept the distance covered so far, both methods learned to sprint and then dive: 3 of 3 PPO policies and 2 of 3 CMA-ES gaits ended their trial by tipping. Only when tipping cost the whole run (score −1 m) did they keep the body level.

Is PPO better than CMA-ES once the reward is fixed?

With the forfeit rule and 20 slightly randomized starts per controller, PPO averaged 4.0 m and CMA-ES 2.8 m (3 seeds each). PPO was faster but less consistent: 23 of its 60 runs tipped versus 15 of 60 for CMA-ES, and 92 of its 300 checkpoint evaluations after 2,400 trials scored −1.

Why not just add a tilt penalty to the reward?

You can, and many environments do (posture, energy and survival terms). We tested only rules that keep the score in metres so CMA-ES and PPO optimize the same number. A penalty weight is one more knob that can be gamed if it's too small or can stop the robot moving if it's too big.

Sources

  1. E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, IROS 2012.
  2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
  3. N. Hansen, The CMA Evolution Strategy: A Tutorial, arXiv:1604.00772.
  4. V. Krakovna et al., Specification gaming: the flip side of AI ingenuity, DeepMind, April 21, 2020.
  5. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané, Concrete Problems in AI Safety, arXiv:1606.06565, 2016 (section on reward hacking).
  6. Farama Foundation, Gymnasium Ant environment (healthy_reward, terminate_when_unhealthy, healthy_z_range).
  7. A. Raffin et al., Stable-Baselines3: Reliable Reinforcement Learning Implementations, JMLR 22(268), 2021.
  8. Experiment, code and measurements: TinyBot Forge, November 2026. Every number on this page comes from the code in the download.

More: CMA-ES vs PPO, the first test · make a robot teach itself to walk · all guides