A reinforcement learning agent trained in a simulator maximizes whatever reward it is given, and it does so under that simulator's physics. When the reward contains a term the simulator answers wrongly, such as an object that slides too easily, a contact that pushes back too hard, or a cube that flips when it should glide, the agent learns to harvest that error. This failure is a kind of reward hacking: the agent scores highly on a number while missing what the designer wanted. It is also the main reason simulator-trained robot policies fail on real hardware. The problem gets worse when the reward is written by a search procedure that is itself scored in the simulator, as with large language models that write reward code. That search selects for reward terms the engine can be fooled into paying out.
We propose to detect this hacking by replaying each policy's own transitions in several independently written physics engines. We record where those engines disagree about what happens next, and we tell the reward designer which reward terms earn their value in exactly those places. The disagreement signal is computed per transition and attributed to individual reward terms. This makes it different in kind from averaging returns over engines or taking the worst engine's return, which only report a scalar. Our claim is that this attribution lets an automated reward designer rewrite the offending terms. The rewritten rewards should transfer to a held-out engine and to a real robot better than rewards chosen by mean or worst-case multi-engine fitness, at a comparable cost in simulated task performance.
Core Claim
The reward hacking literature treats hacking as a gap between a proxy and a true reward under fixed dynamics. Defining and Characterizing Reward Hacking formalizes a proxy as hackable when some policy pair is ranked one way by the proxy and the opposite way by the true reward. It proves that on rich policy sets any simplified proxy is hackable, while unhackable proxies can exist on restricted policy sets. The sim-to-real literature instead treats the gap as a mismatch in transition dynamics. We combine the two views by taking the proxy to be "simulated return under engine $k$" and the true objective to be "task success under real physics." On that reading, a reward is dynamics-hackable exactly when its value depends on transitions the engine gets wrong. We cannot observe those transitions directly. However, Quantifying the Reality Gap in Robotic Manipulation Tasks shows that engines agree closely with motion capture in free space and diverge sharply in contact, for instance on a 90° cube pitch in MuJoCo and Bullet that reality does not show. Independent engines therefore tend to disagree precisely where any one of them is unreliable.
Concretely, for a policy $\pi$ trained in engine $k$, we sample visited states $s$ and actions $a$, set the same rigid-body state in each of $M$ engines, step once (or a short window of $H$ steps), and compute
$$\Delta(s,a) = \frac{1}{M}\sum_{m=1}^{M}\left\lVert \phi(s'_m) - \bar\phi \right\rVert^2,$$
where $s'_m$ is the next state in engine $m$, $\phi$ extracts task-relevant quantities (object pose, contact forces, joint velocities), and $\bar\phi$ is their mean across engines. For a reward written as a sum of terms $r_i$, we define the term's exposure as the fraction of its accumulated value earned on transitions where $\Delta$ exceeds a noise floor $\delta$:
$$E_i(\pi) = \frac{\mathbb{E}_{d^\pi_k}\left[\,|r_i(s,a)|\,\mathbf{1}[\Delta(s,a) > \delta]\,\right]}{\mathbb{E}_{d^\pi_k}\left[\,|r_i(s,a)|\,\right]},$$
with $d^\pi_k$ the policy's state-action visitation in engine $k$. We calibrate $\delta$ on free-space motions, where the reality-gap measurements say engines should agree. Exposure scores serve two roles. They are fed back as text to a language-model reward writer, and they are added as a penalty $-\lambda \Delta$ during policy training.
Research Program
The program has three parts. The first is infrastructure. We build a state-transfer harness covering the engines used by PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization (IsaacGym, IsaacSim, Genesis), plus SAPIEN for manipulation tasks. MuJoCo is held out as an unseen engine. Rigid-body poses, velocities, and joint states transfer directly. Contact state does not transfer. We handle this with a short settling window and by measuring the residual disagreement it introduces on no-contact transitions, which sets $\delta$. Massively parallel simulators such as Isaac Gym make replaying a subsample of transitions cheap compared with training.
The second part is the diagnostic study. Before changing any method, we ask whether exposure predicts transfer failure. We take reward functions produced by an Eureka-style loop, which writes reward code with a language model, trains policies in Isaac Gym, and evolves rewards from training-curve reflections. We also take the human-written rewards on the same tasks. For each reward we train policies and measure the drop in success when moving to MuJoCo and, for manipulation, to a real arm. We then test whether policy-level exposure correlates with that drop more strongly than simpler predictors do. The simpler predictors are the variance of return across training engines and the gap between mean and minimum engine return. Eureka's authors already identify sim-to-real as open because generated rewards exploit simulation-specific contact behavior. That observation motivates the study, but no transfer numbers for Eureka rewards appear in our file, so the study has to produce them.
The third part is the method. In the reward-design loop, each candidate's reflection gains a per-term exposure report, for example "the grasp-contact term earned 71% of its value on transitions where engines disagree on object velocity." Selection uses held-in mean fitness subject to a cap on total exposure. During policy training we add the $\Delta$ penalty, estimated on a replayed subsample.
The method borrows ideas from several prior works while differing from each. ERMAS (ERMAS) also frames a "reward sim-to-real gap", but for other agents' utilities in economic games rather than physics. Preventing Reward Hacking with Occupancy Measure Regularization regularizes state visitation toward a reference policy. That reference lives in the same flawed simulator, whereas our penalty needs no reference policy and targets regions where physics is unreliable rather than regions that are merely unfamiliar. A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation tolerates misspecified low-fidelity rewards by using them as variance reducers for high-fidelity gradients. That approach requires high-fidelity rollouts, which reality cannot supply at scale, and it is limited to REINFORCE.
Baselines
The naive reference point is Eureka run in a single engine (IsaacGym), with fitness equal to simulated task success, followed by direct deployment to MuJoCo and to hardware. Alongside it we run the human-engineered rewards trained with PPO in the same engine. These two show how much transfer collapses with no protection.
The strongest existing competitor is multi-engine training in the style of PolySim. Policies are trained on environments spread across IsaacGym, IsaacSim, and Genesis, and they maximize mean return. On the ASAP humanoid motions, PolySim reports a 52.8% higher execution success rate in unseen MuJoCo than IsaacSim-only training. We pair this with Eureka using mean multi-engine fitness, so that both the reward search and the policy see all engines. A second strong competitor is worst-case selection, in which Eureka uses the minimum fitness across engines and policies optimize a soft-minimum of engine returns. This tests whether a scalar robustness criterion already captures what exposure captures.
To compare the physics-side alternatives, we run uniform domain randomization over mass, friction, and damping. We also run EASI (EASI), which identifies simulator parameters from a single real trajectory using evolutionary search and a transition discriminator, and reports roughly 37% lower pole angle error than uniform randomization on real Cartpole. To compare the reward-side alternative, we run ORPO-style occupancy regularization toward the policy trained on the human reward.
Every arm gets the same number of language-model reward iterations, the same total environment steps (counting replay steps against our method's budget), and the same hyperparameter search. The ablations that isolate our ingredient are three. The first passes scalar total disagreement to the reward writer without per-term attribution. The second keeps the training penalty but drops the attribution feedback. The third keeps the feedback but drops the penalty. We count the proposal as beating the field if it improves held-out-engine and real success over both the mean and the worst-case multi-engine arms. Held-in simulated success may fall, but the fall must stay within a stated margin, which we fix at five points before running. The method should also beat the scalar-disagreement ablation, since that ablation isolates the effect of attribution.
Why not just…?
Why not just train across multiple engines, as PolySim does? Averaging returns rewards a policy for any engine's quirk that does not actively hurt it in the others. It also leaves the reward function untouched, and PolySim's own limitation is that it inherits whatever its hand-written tracking reward gets wrong. Our comparison against mean multi-engine Eureka settles whether this matters. If that arm matches ours on MuJoCo and hardware, the attribution adds nothing, and we will report that.
Why not just take the worst engine's return? This alternative is the most serious threat. Worst-case selection removes an exploit present in only one engine, and it is simpler. Its weakness is that it tells the reward writer that a reward is fragile but not which term is fragile or why. It also says nothing on transitions where engines happen to agree on reward but disagree on state. The scalar-disagreement ablation and the worst-case arm together decide the question. If worst-case selection matches us, the contribution reduces to the diagnostic study, and the method claim is withdrawn.
Why not just randomize physics parameters or identify them from real data? Parameter randomization cannot change how an engine resolves contact. PolySim reports beating parameter-level randomization on unseen physics, and EASI notes that wide randomization yields sluggish, overly conservative behavior. System identification aligns parameters but not solver structure. These methods also complement ours, so we additionally run our method on top of EASI-identified parameters.
Why not just use a vision-language model as the judge? The VLM observer in RoboTwin 2.0 scored 0.431 accuracy and could not see grasp-axis errors. PRISM needed geometric multi-view prompts to make such judging useful. Contact exploits are largely invisible to cameras, which is why we measure dynamics directly.
Proposed Experiments
The simulated suite uses Eureka's Isaac tasks and its Shadow Hand dexterity tasks, where the original reports beating human rewards on 83% of 29 tasks. For these tasks we found no reported result on Lacuna for transfer to another engine, so the MuJoCo-held-out success of Eureka rewards is a number we will measure. We add the ASAP humanoid motions with MuJoCo held out, using PolySim's 52.8% relative gain as the reference. We also add a contact-heavy cube push modeled on the reality-gap testbed, where we expect exposure to concentrate. For hardware, we use ManiSkill2 PickCube, where a simulation-trained policy reached 91% in simulation and 60% zero-shot on a real robot, and a planar push task on the same arm. For broader manipulation evaluation we use REALM. Its 0.921 correlation between simulated and real progression lets its system-identified simulator serve as an additional held-out reality proxy, scored with its tiered progression rubric.
There are three decisive checks. First, across all trained policies, exposure must predict the held-out and real success drop with a higher rank correlation than return variance across engines; if it does not, the premise fails. Second, in a controlled injection test, we plant a reward term known to pay out on an engine artifact, such as rewarding cube angular velocity during a push. The attribution must flag that term, and the language-model writer must remove or rewrite it within the iteration budget. Third, on PickCube hardware, the method must exceed both the 60% zero-shot reference and the worst-case multi-engine arm, while its held-in success stays within the pre-registered margin.
References
- Defining and Characterizing Reward Gaming
- Quantifying the Reality Gap in Robotic Manipulation Tasks
- PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Eureka: Human-Level Reward Design via Coding Large Language Models
- ERMAS: Becoming Robust to Reward Function Sim-to-Real Gaps in Multi-Agent Simulations
- Preventing Reward Hacking with Occupancy Measure Regularization
- A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
- EASI: Evolutionary Adversarial Simulator Identification for Sim-to-Real Transfer
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation