Is LLM Reward Design Reasoning or Recall?

Large language models can now write reward functions, the scalar scoring code that tells a reinforcement learning agent what to do. Eureka reports that GPT-4, given an environment's source code and a loop of trial training runs, beats expert-written rewards on 83% of 29 simulated robotics tasks. That result matters because hand-writing rewards is slow and error-prone. In one study, 17 of 18 self-identified experts produced misaligned rewards even in a 4x4 grid-world. Every environment in that evaluation, however, is public code on GitHub, usually shipped alongside its human-written reward. Work on benchmark contamination shows that models score far higher on test material they have seen during pretraining. Qwen2.5-Math-7B, for example, solves nearly 40% of MATH-500 from only the first 40% of each problem. The same could be true here: an LLM reward designer may be partly retrieving reward templates it has already read.

This note proposes a contamination audit for automated reward design, followed by a test of what the audit implies for reward hacking. Reward hacking means an agent maximizing the score while breaking the intent behind it. We will measure how much of the LLM's advantage over human experts survives when the environment cannot be recalled. We expect memorized templates to carry hidden safeguards that human engineers added over years, such as torque penalties and stability terms. If so, rewards written for truly novel environments should hit the task metric just as well while producing more degenerate behavior. If that holds, the field's evidence that LLMs solve the reward engineering bottleneck is weaker than it looks. It would also mean the useful mitigation is a cheap ranking-based alignment check, not more LLM iterations.

Core Claim

Our claim is that LLM reward synthesis on public benchmarks is inflated by recall, and that the inflation hides a reward-hacking cost. We make three testable predictions. First, a per-task memorization score, measured by asking the model to finish a truncated environment file, will predict how far Eureka-style rewards beat human rewards on that task. Second, on procedurally generated environments with no public source, the win rate against human experts will drop. Third, and most important, behaviors the reward designer never saw scored will degrade more than the selection metric does. These include joint-limit violations, energy use, oscillation, and self-collision.

The third prediction separates this work from a plain "LLMs memorize benchmarks" result. Eureka selects candidate rewards with a ground-truth fitness function, so success on that fitness is guaranteed to look good. Countdown-Code defines hacking as the gap between a proxy reward and the true objective. Following it, we hold out constraint metrics as the true objective and read hacking off the gap between them and the fitness the designer optimized.

Relation to Prior Work

The proposal is motivated by two contamination papers that work only in symbolic domains. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination shows that reinforcement learning (RL) gains on MATH-500, even under random rewards, disappear on its uncontaminated RandomCalculation generator. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks shows that models strong on static benchmarks score near zero on graph-generated reasoning tasks. We borrow the partial-prompt audit from the first paper and the "generate fresh instances" logic from the second. We move both from evaluating answers to evaluating reward code for physical control, a domain the research file notes procedural evaluation has not reached.

The systems under audit are Eureka and CoRe. Eureka is the evolutionary LLM reward coder evaluated on the Isaac suite. CoRe pairs LLM-generated code rewards with residual rewards learned from vision-language model preferences, and accepts code only if it agrees with those preferences; it reports 99.0% average success on Meta-World. Neither paper checks whether its generator or judge relies on familiarity with the benchmark. We do not propose a better reward generator. We propose a test of whether the existing ones generalize.

The closest pairwise work is Countdown-Code. It shows that 1.2% contaminated cheating demonstrations in supervised fine-tuning seed reward hacking, which RL then amplifies. Its contamination is in the policy's training data and its reward is a fixed binary harness. Ours is pretraining exposure of the reward designer, with shaped continuous rewards.

CARMO argues that any fixed set of evaluation criteria leaves some true objective invisible. It answers this with criteria generated per query, but only for chat alignment. Its argument motivates our use of hidden constraint metrics that the designer never sees.

For the mitigation arm, the method comes from Analyzing Reward Functions via Trajectory Alignment and The Trajectory Alignment Coefficient in Two Acts. The Trajectory Alignment Coefficient (TAC) is the Kendall rank correlation between how a candidate reward orders a set of trajectories and how a stakeholder orders them. It lifted Lunar Lander tuning success from 19.8% to 62.7% when given to practitioners. Those papers apply TAC to human tuning on fixed environments. We apply it as an automatic filter on LLM-generated candidates for unseen environments.

B-Pref constrains the design. Because noisy teachers break preference methods, the rankings fed to TAC must be tested under simulated label noise, not only clean oracle labels.

Research Program

The first component is a memorization audit. For each Isaac Gym and Meta-World task, we truncate the public environment file at 40%, 60%, and 80% of its length with the reward function removed. We then ask the reward-designer model to write the reward. We score structural similarity (abstract-syntax-tree overlap of reward terms and their coefficients) to the reward in the public repository. This yields a per-task memorization score, analogous to the partial-prompt test of Wu et al.

The second component is a ladder of three environment conditions with increasing distance from pretraining data. The canonical condition is the public environment as released. The obfuscated condition keeps the same physics but renames all variables, permutes the observation vector, and rescales units. The procedural condition uses a new generator built on Isaac Gym's GPU simulation. It samples articulated morphologies (link count, joint axes, actuator limits) and composes task goals (reach, carry, reorient, balance under perturbation) with freshly generated identifiers. We validate each generated environment with the same audit; it should show near-zero memorization. We also check that a sparse-reward agent's success rate falls in the same range as on the canonical tasks, so the procedural tasks are not simply harder.

The third component is a measure of hacking. Every environment carries hidden constraint metrics that the reward designer and its fitness loop never see: joint-limit violation rate, mechanical energy per unit progress, high-frequency action oscillation, and self-collision count. Following Countdown-Code, hacking is the divergence between fitness achieved and these held-out quantities. As an independent check, three annotators who do not know the condition will rank 20 short rollout videos per policy for naturalness.

The fourth component is the mitigation test. We collect a small set of trajectory rankings per novel environment, about 30 pairs from an annotator, with a noise-injected version following B-Pref. Each Eureka candidate's TAC against these rankings joins the scalar fitness as a selection criterion. We compare this with CoRe's preference-agreement filter at matched label counts.

Baselines

The naive reference point is a sparse task-success reward with the same RL algorithm and training budget. It anchors the normalized score used by Eureka, in which 0 means sparse-reward performance and 1 means human-expert performance.

The human-expert reference on canonical tasks is the reward shipped with each environment. On procedural tasks, no human reward exists. We will recruit RL practitioners and give each a fixed four-hour budget per task with access to training curves. This is roughly the setting where Muslimani et al. found 94% misalignment, which makes it a realistic rather than idealized human baseline.

The strongest existing method is Eureka. It reports beating human rewards on 83% of 29 tasks with a 52% average normalized improvement (Eureka). We run it with the same number of evolutionary rounds, candidates per round, and RL steps per candidate in all three conditions, using the same base model. Separating recall from reasoning requires a fixed base model, so we report Eureka's published numbers rather than claiming to reproduce them.

On Meta-World, the strongest preference-aligned competitor is CoRe at 99.0% average success with 0.15K–0.5K labels (CoRe). We run it with its label budget matched to the TAC filter's. The research file reports no CoRe results on held-out or obfuscated environments.

Two ablations isolate the key ingredients. The first isolates environment-code context: Eureka given only a natural-language task description and the observation schema, with no source code. The second isolates the alignment filter: TAC selection against Eureka's scalar fitness alone.

The proposal's empirical claim is confirmed if two things hold. Eureka's normalized advantage must fall by a statistically clear margin from the canonical to the procedural condition, while its own selection fitness stays comparable. And hidden-constraint violations must rise. The mitigation beats Eureka if it lowers hidden-constraint violations on procedural environments without losing more than a few points of task success. It beats CoRe if it does so at equal labels.

Why not just…?

Why not just obfuscate variable names and skip the procedural generator? Obfuscation is the cheapest contamination control, which is why it is a condition rather than an afterthought. It is a real threat to the procedural generator's value. If the obfuscated and procedural conditions give the same win rates and hacking rates, we will conclude that surface recall drives the effect and that renaming is a sufficient audit tool. We would report that as a finding. Our expectation is that renaming leaves the physics, and therefore the familiar reward structure (for example, "forward velocity minus torque cost" for a quadruped), recognizable. Only new morphologies and task compositions remove it.

Why not just measure code similarity between Eureka's outputs and the public rewards? Similarity shows overlap but not dependence. A model could independently rediscover "velocity minus action penalty" because it is a sensible design. The partial-file completion audit asks whether the model can produce the specific repository reward without seeing it. The procedural condition then tests whether performance depends on that ability. Similarity scores will be reported, but only as a secondary correlate.

Why not just evaluate on environments released after the model's training cutoff? Post-cutoff environments are a valid natural experiment, and we will include any suitable ones. They are few and uncontrolled in difficulty, though. They are also often built from existing templates, so their reward structure may still be familiar. DyVal's lesson is that controllable generation is what makes contamination effects measurable. Fresh-but-uncontrolled tasks cannot separate novelty from difficulty.

Why not just tune Eureka's fitness function to include the constraint terms? This works only if one already knows which constraints the agent will violate. That is exactly the hand-engineering burden, and the misalignment rate, that the TAC studies document. CARMO's argument that any fixed criteria set leaves exploitable gaps applies directly. Still, we will run a variant that adds generic energy and joint-limit penalties to the fitness. If it closes the hacking gap on procedural tasks as well as TAC does, the cheaper fix wins and we will say so.

Proposed Experiments and Decisive Checks

The first decisive check is correlational, across about 40 canonical tasks from the Isaac suite and Meta-World. Does the per-task memorization score predict Eureka's per-task normalized advantage over the human reward? A positive rank correlation that holds across two base models would show that recall is doing measurable work.

The second check is the condition ladder. We will run 20 procedural tasks, matched in sparse-reward difficulty, against their canonical and obfuscated counterparts, with five seeds each. We compare win rate against human rewards, selection fitness, and the four hidden-constraint metrics. The pattern that would most clearly confirm the hypothesis is stable fitness, a falling win rate, and rising constraint violations. Null results would undercut it and are publishable in their own right. Null results here are flat win rates and flat violations from canonical to procedural, meaning LLM reward design generalizes as claimed.

The third check is the mitigation comparison on procedural tasks. We compare TAC selection with 30 ranked pairs, CoRe's filter at matched labels, and the fitness-penalty variant, each under clean and noise-injected rankings. The primary metric is hidden-constraint violation rate at a fixed task-success threshold, with blinded human naturalness rankings as confirmation. A last sanity check validates the hidden metrics themselves: on canonical tasks, human-expert rewards should show low violation rates. If they do not, the metrics are measuring something other than hacking and must be revised before any conclusion is drawn.