When people train a reinforcement learning (RL) agent, they rarely write the reward function once. They write a candidate, train an agent, watch what it does on the training environments, adjust the shaping terms, and repeat until the agent looks good. This loop is a hyperparameter search in which the "hyperparameter" is the task objective, and the score that guides it comes from the same levels the agent trains on. Supervised learning learned long ago that tuning on the training set inflates results. RL reward design still routinely tunes this way. We propose to measure how much of the generalization gap of deep RL agents comes from the selection of the reward rather than from the policy network memorizing levels, and to test a simple remedy: choose rewards on held-out levels.
The two literatures we connect have each studied half of this. Reward design research showed that hand-tuned rewards overfit to the learning algorithm they were tuned with. Generalization research showed that value networks memorize training levels. Nobody has asked whether reward functions themselves overfit to training levels. Our hypothesis is that the trial-and-error loop systematically favors shaping terms that happen to make the training levels easy, and that this produces an extra, predictable drop in held-out success. That drop grows with the number of candidate rewards tried, and no policy-side regularizer can remove it.
Core Claim
We define a reward-selection gap. It is the drop from training-level success to held-out-level success for the reward a designer actually picks, minus the same drop for a reward drawn at random from the designer's candidate pool. The random-draw gap measures ordinary policy overfitting. The excess measures overfitting caused by selecting the reward.
We claim three things. First, the selection gap is positive and grows with the number of candidates evaluated and with the number of free shaping weights. Second, it is concentrated in shaping terms that are not potential-based, meaning terms that change which policy is optimal rather than only how fast it is found, and in terms keyed to layout-specific quantities such as straight-line distance to a goal in a particular maze. Third, selecting on a held-out split of training levels removes most of the gap at no extra training compute.
The closest prior work is The Perils of Trial-and-Error Reward Design. It swept 5,196 reward functions in a tabular gridworld and found that reward rankings barely transfer across algorithms (Kendall's τ between PPO and DDQN of −0.04). That paper treats overfitting as overfitting to the solver, in a single environment with no neural function approximation. We hold the solver fixed and vary the environment instance. This is a different axis of overfitting, and it only appears in procedurally generated, multi-instance settings.
Research Program
The program has four parts.
Reward families. We build a parameterized family of rewards for each environment. Each family is a weighted sum of components a practitioner would plausibly write. In Procgen (Leveraging Procedural Generation to Benchmark Reinforcement Learning) we use Maze, Heist, CoinRun and CaveFlyer, because each has an obvious shaping proxy. The components are:
- straight-line or path distance to the goal or key,
- progress along a direction,
- a time penalty,
- subgoal bonuses,
- a potential-based version of each distance term, of the form $F = \gamma\Phi(s') - \Phi(s)$.
Shaping terms of this last form leave the optimal policy unchanged. Shaping signals are computed from simulator state, never from pixels, so the reward cannot directly read distractors. The agent sees only the standard 64×64 images.
In Meta-World (Meta-World) we take the reach, grasp and place components the authors hand-built. We let their weights and stage thresholds vary, and treat object and goal positions as the "levels."
Simulated trial-and-error designer. The designer is an automated search over the reward family, run with a fixed budget of K candidates (K from 4 to 64). Its only feedback is true task success on its tuning levels. We pick the best candidate, retrain with fresh seeds, and evaluate on both the tuning levels and the full level distribution.
Retraining with fresh seeds separates two sources of optimism:
- Seed-noise winner's curse: the best of K noisy scores is optimistic even on the same levels.
- Level overfitting: the chosen reward truly fits those particular levels.
We run three variants of the search. One tunes on the agent's own 200 training levels, which is common practice. One splits those 200 levels into 150 for training and 50 for selection. One tunes on all 200 levels but uses extra seeds, which matches the compute of the split variant.
Small human replication. We will run a human study modeled on the Booth et al. protocol, with about a dozen RL practitioners and a restricted budget of design iterations on Procgen Maze. This checks that real designers produce selection gaps comparable to the automated search. Booth et al.'s finding that 97% of experts add shaping suggests they will reach for the layout-keyed terms we hypothesize are risky.
Mechanism. For selected versus randomly drawn rewards, we measure two properties of the value network (the critic):
- Update rank, from Learning Dynamics and Generalization in Deep Reinforcement Learning. That paper showed TD learning, the bootstrapped regression used to train value networks, fits jagged components of the value function first, and that rising update rank tracks memorization.
- Stiffness, from Rethinking Value Function Learning for Generalization in Reinforcement Learning. Stiffness measures how much an update at one state moves predictions at similar states.
Both papers held the reward fixed. We ask whether the selection loop picks rewards whose value landscapes are easier to memorize.
We also test the concern raised by Unpacking Reward Shaping. That paper proved shaping speeds learning by steering exploration away from states it marks as unpromising, in a single tabular MDP. In procedurally generated levels, a region that is a dead end in training layouts may be the only route in a test layout. We test this by measuring held-out-level state coverage and the failure modes of the selected agents.
Baselines
Naive reference points. The first is the default environment reward with no search: the Procgen game reward, and the original Meta-World dense reward. The second is the pure sparse success reward. Neither involves selection, so both have zero selection gap by construction. Their held-out success is the bar any tuned reward must clear to be worth tuning. Our measurement only matters if tuned rewards typically win on training levels and lose part of that advantage on held-out ones.
Naive trial-and-error selection on training levels. This is the practice under study. It also serves as a baseline for the remedy.
Strongest existing generalization method. This is DCPG from Rethinking Value Function Learning for Generalization in Reinforcement Learning. It updates the critic less often from a large replay buffer. That paper reports a PPO-normalized test return of 184.5% across the 16 Procgen games, against 160.3% for Phasic Policy Gradient, and 202.2% for the dynamics-aware variant.
We will run DCPG with the authors' hyperparameters on our four games, in the 200-level easy-mode regime. Every reward condition gets identical environment steps and seeds. The key question is whether DCPG combined with a naively selected reward still shows a selection gap. If it does, policy-side fixes and reward-side selection are complementary. If DCPG with naive selection matches PPO with held-out selection, our remedy has no practical value on Procgen and we will say so.
Strongest Meta-World reference. This is multi-task SAC with the original rewards. Meta-World reports 85% success on MT10, 40% on MT50 and 30% held-out success for PEARL on ML45. We use the MT10 setting with held-out object and goal positions. No reported result was found on Lacuna for held-out-position success under tuned reward weights.
Automatic shaping without a human. These are BSRS from Bootstrapped Reward Shaping and SASR from Highly Efficient Self-Adaptive Reward Shaping. BSRS uses the agent's own value estimate as the shaping potential and reports a 45–60% human-normalized gain on Atari. SASR shapes from visitation success rates. Both remove most hand design, but their own scale parameters are still tuned on training levels. We give each the same K-candidate budget over its scale parameter.
Ablations.
- Compute-matched same-level search. We search on the same levels with extra seeds at equal compute. This isolates whether the benefit comes from level novelty or merely from less noisy selection.
- Restricted shaping family. We restrict the family to potential-based terms only. This isolates whether non-invariant shaping drives the gap.
The remedy "beats" naive selection only if all three of these hold: its held-out success is higher by more than the seed-to-seed spread across five seeds; the compute-matched same-level ablation does not close the difference; and the advantage grows with K.
Why not just…?
Why not just use potential-based shaping, which provably cannot change the optimal policy? Invariance is a statement about optimal policies, not about what a finite-budget neural agent actually learns. A potential keyed to maze distance can still reshape value targets and exploration in ways that fit the training layouts. The potential-only arm settles this directly. If its selection gap is near zero, the practical recommendation becomes "restrict to potentials" rather than "select on held-out levels," and we will report that.
Why not just use sparse rewards and avoid design altogether? On games where sparse PPO learns, this is a real competitor, and the sparse baseline tests it. But the file's evidence is that unshaped learning is often slow or fails. Bootstrapped Reward Shaping reports a 45–60% human-normalized deficit for the unshaped setting on Atari. Unpacking Reward Shaping shows that without shaping, exploration cost grows with the full state space. Practitioners will keep shaping, so the question is how to select shaping honestly.
Why not just regularize the policy (DCPG, distillation, weight decay)? These methods fix how the network fits a given reward. They cannot undo the choice of reward. Distillation in Lyle et al. copies whatever behavior the teacher learned. Efficient Deep Reinforcement Learning Requires Regulating Overfitting selects regularizers by held-out TD error, which is error measured on the shaped reward and is therefore blind to a reward that fits the training levels. The DCPG-with-naive-selection arm is the empirical test, and it is the main threat to the proposal.
Why not just train on more levels? Procgen needs up to 10,000 levels before test performance catches up with training performance (Leveraging Procedural Generation to Benchmark Reinforcement Learning). With that many levels, the selection gap should shrink too. We will run one game at 10,000 levels to check. Most real tasks, however, have a handful of training instances, not thousands.
Proposed Experiments and Decisive Checks
Main experiment. The main experiment crosses four Procgen games and Meta-World MT10 with:
- search budgets of 4, 16 and 64 candidates,
- three selection protocols (train-level, held-out-level, compute-matched same-level),
- two shaping families (full, potential-only),
- five seeds per selected reward.
Training uses PPO, plus DCPG on Procgen, at 25M environment steps in easy mode. The search costs roughly 800 Procgen runs, which is why we keep to four games.
Metrics. On Procgen, the primary metric is true task success, not shaped return, on the full level distribution. On Meta-World, it is success on held-out positions. The secondary metric is the reward-selection gap as defined above. Mechanistic metrics are update rank, stiffness and held-out-level state coverage.
Decisive checks. The first check is whether the selection gap is positive and increasing in K under training-level selection. If it is flat at zero, the thesis is wrong and the paper becomes a negative result about reward design that still informs practice. The second check is whether held-out-level selection beats the compute-matched same-level ablation. If it does not, the effect is seed-noise winner's curse rather than level overfitting. The third check is whether the gap shrinks in the potential-only family and at 10,000 levels, which the mechanism predicts. The fourth check is whether selected rewards show higher critic update rank and lower stiffness than randomly drawn rewards of equal training success. That pattern would tie selection to the memorization dynamics described by Lyle et al. and Moon et al. The human study then checks that the automated designer is a fair stand-in for people.
References
- The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Learning Dynamics and Generalization in Deep Reinforcement Learning
- Rethinking Value Function Learning for Generalization in Reinforcement Learning
- Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity
- Bootstrapped Reward Shaping
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning
- Efficient Deep Reinforcement Learning Requires Regulating Overfitting