Anti-Goal Mutation for LLM Reward Search

Large language models can now write reward functions for robots. They write candidate reward code, train a policy on each candidate, read back training statistics, and rewrite the best one. This loop beats hand-written rewards on most simulated tasks. It inherits a known weakness of greedy evolutionary search, though. When every candidate reward leads the robot into the same bad behavior, such as pushing against a wall or locking a grasp too early, the loop keeps polishing that behavior instead of leaving it. We propose a different way to generate new reward candidates when search stalls. First, look at where the stuck policies actually end up in state space. Second, have the language model write that region into the reward code as an explicit "anti-goal" penalty. Third, keep the resulting candidates alive under non-elitist selection long enough to cross the temporary drop in task score.

Put simply, the proposal turns the anti-goal trick that Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards applied inside a single RL run into a mutation operator over reward programs. It pairs that operator with the selection rule that theory says is needed to escape fitness valleys. The question we test is narrow and falsifiable. On tasks where LLM reward search plateaus, does trap-aware mutation with tolerant selection escape more often than spending the same compute on independent restarts?

Core Claim

Reward-code search fails in a specific way. Its candidates differ in syntax but converge on the same behavioral attractor, a region of state space that the obvious shaped rewards all point toward. Two ingredients are needed to leave such an attractor, and current systems have neither.

The first ingredient is a mutation that knows where the trap is. Classical stochastic mutation, whether by uniform re-initialization (A scalable coevolutionary multi-objective particle swarm optimizer) or small Gaussian perturbation (Human Activity Discovery With Automatic Multi-Objective Particle Swarm Optimization Clustering With Gaussian Mutation and Game Theory), is blind to why a region is a trap. In high dimensions, blind jumps rarely find an exit. The second ingredient is a selection rule that tolerates a short-term loss. Escaping Local Optima with Non-Elitist Evolutionary Algorithms proves that elitist $(\mu+\lambda)$ selection needs $n^{\Omega(n)}$ time to cross the valley in the FUNNEL function, while 3-tournament selection crosses it in polynomial time. The same paper shows an "error threshold": mutation that is too strong erases what the population has learned. Our design responds to both results. The anti-goal mutation is strong in one targeted direction and leaves the rest of the reward intact, so it stays below that threshold.

Research Program

The base loop follows Eureka: Human-Level Reward Design via Coding Large Language Models. Eureka gives environment source code to an LLM, samples reward programs, trains them in parallel in Isaac Gym, and asks the LLM to revise the best one using a text summary of training statistics. We also reuse two pieces of LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search. The first is its shared replay buffer that relabels old transitions under new rewards. The second is its option to warm-start policies from elites. Both are there to limit the cost of each generation.

We add three components. The first is a stagnation detector. Every candidate's final policy is summarized by a behavior descriptor: the distribution of terminal and late-episode values of task-relevant state variables, such as gripper pose relative to the handle, base position in the maze, or object height. These descriptors are clustered across the population. Search counts as stalled when two things hold at once. Task fitness has not improved for a set number of generations, and most candidates' descriptors fall in one cluster. This follows the observation in A Mutation Operator Accelerated Quantum-Behaved Particle Swarm Optimization Algorithm for Hyperspectral Endmember Extraction that mutation matters most once diversity has decayed late in a run.

The second component is anti-goal mutation, which fires only when the detector does. The dominant cluster's centroid and spread are given to the LLM in plain terms, together with the elite reward code. An example: "policies end with the gripper 2 cm left of the handle, fingers closed, drawer unmoved." The LLM must add a term that penalizes ending near that configuration while keeping the elite's other components. The penalty magnitude is not chosen by the LLM. It is set as a fraction of the elite reward's typical return scale and drawn from a small schedule across offspring. Different offspring therefore probe different mutation strengths, which turns the narrow sweet spot from Dang et al. into something the population searches over rather than something we hand-tune. Sibling Rivalry builds anti-goals from paired rollouts that start from the same reset state, and it does so for each policy update. Our version works differently in three ways. It is built from the population's history, so it needs no paired resets. It lives in the reward code, so it persists across generations. It is expressed in task variables the LLM can generalize semantically, rather than as raw coordinates.

The third component is tolerant selection with a bounded escape window. Anti-goal offspring compete with one another in 3-tournaments rather than against the elite. They survive for a fixed number of generations even if their task fitness is lower. If no descendant beats the old elite within that window, they are dropped. During the window, policies warm-start from the elite with relabeled data. We turn off LaRes's parameter-closeness loss, because that loss anchors the policy to exactly the behavior we want it to leave. This is the tension between escaping and avoiding policy collapse that the research file identifies as unresolved, and we measure it directly.

The analysis has two parts. One checks whether the measured escape behavior fits the FUNNEL picture: whether elitist variants show escape times that grow sharply with trap depth and tournament variants do not. The other checks how often the LLM actually encodes the anti-goal faithfully in code. We audit this by checking whether trained policies' descriptors move away from the targeted cluster.

Baselines

The do-nothing reference has two parts. One is the task's human-written reward from the benchmark suite. The other is vanilla Eureka run to the same budget. Eureka is the system whose elitist revision we claim stalls. It is run with its published population size and iteration scheme, and its total RL environment steps and LLM calls are matched to ours.

The strongest existing methods differ by benchmark. On Isaac Gym manipulation it is R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models. R breaks rewards into modules, recombines modules across parents, and fits the continuous weights from preference votes by LLM-written critics. It reports a 73% success rate on FrankaCabinet against Eureka's 33% under identical compute. That makes it the bar on the task where search headroom is clearest. On MetaWorld and MuJoCo locomotion the bar is LaRes. LaRes reports near-100% success on Hammer and Basketball, where baselines stay below 40%, and 4.44 m/s on Ant from scratch versus Eureka's 1.49 m/s. Those MetaWorld numbers leave little headroom, so we treat them as a non-regression check rather than a target. We found no reported result on Lacuna for LaRes on FrankaCabinet or for R on MetaWorld, so we run both systems on both suites ourselves with their released settings and the same budget. For the visual-feedback line we include Evolutionary Reward Design and Optimization with Multimodal Large Language Models. EROM adds rendered videos to the LLM's reflection, and it reports Ant 7.27 versus Eureka 3.68 and Humanoid 5.26 versus 4.21. It matters here because visual critique is the main competing way to tell the LLM what the trap looks like.

On deceptive navigation we use the Sibling Rivalry suite. There, PPO with a naive distance reward fails 100% of the time on Point-Maze and Sibling Rivalry solves it consistently. Sibling Rivalry is the policy-level anti-goal baseline and is given the paired resets it requires.

Ablations isolate the two ingredients. One is anti-goal mutation with elitist selection. Another is tournament selection with ordinary Eureka-style mutation, which uses the same detector and window but no trap description. A third replaces anti-goal mutation with isotropic macro jumps, meaning fresh reward programs sampled from scratch at stagnation. A fourth keeps the parameter-closeness loss on during escape windows.

Beating a baseline means a higher final success rate under matched steps and LLM calls, averaged over at least five seeds, with non-overlapping bootstrap intervals. On FrankaCabinet in particular, beating R* means exceeding its success rate in our own runs, which should land near the reported 73% if the reproduction is faithful.

Why not just…?

Why not just restart Eureka several times and keep the best run? This is the most serious threat. If stagnation comes from sampling variance rather than a shared attractor, independent restarts at matched compute will escape as often as our method, and our claim fails. We run restarts as a first-class baseline. For each task we also report the share of independent Eureka runs that end in the same behavior cluster. If restarts land in different clusters, the attractor story is wrong for that task, and we will report this rather than argue around it.

Why not just keep the top-k candidates or raise the LLM's sampling temperature? Both add syntactic diversity. Our hypothesis is that syntactic diversity does not produce behavioral diversity when every plausible shaped reward points at the same attractor. The tournament-plus-ordinary-mutation ablation is exactly this alternative with a fair selection rule. The behavior-cluster measurements show whether higher temperature actually spreads policies out.

Why not just run Sibling Rivalry or curiosity on top of Eureka's best reward? Sibling Rivalry reports that curiosity bonuses and Hindsight Experience Replay fail on its distance traps, so curiosity alone is a weak candidate. Sibling Rivalry itself is strong where paired resets exist. We include Eureka-best-reward plus Sibling Rivalry as a baseline. If it matches us, the conclusion is that anti-goals belong at the policy level. Our advantages would then reduce to not needing resets and to the anti-goal persisting across generations.

Why not just add noise to the reward, as ReDit does? ReDit: Reward Dithering for Improved LLM Policy Optimization adds zero-mean noise to 0/1 rewards to smooth plateaus in language-model reasoning. Its own ablations report no benefit for continuous rewards, which are what robotics uses. We include it once on Point-Maze as a sanity check, and we expect it to fail.

Proposed Experiments

The main testbed is a set of tasks chosen because simple shaped rewards trap them. We use Sibling Rivalry's U-wall Point-Maze and Ant-Maze, rebuilt in Isaac Gym, with the LLM given only the goal description. This lets us confirm that Eureka itself produces distance-style rewards. We vary wall depth to create a family of trap depths, a rough physical analogue of the FUNNEL function. FrankaCabinet and two R* dexterous tasks, Shadow-Hand-Kettle and Allegro-Hand, form the realistic headroom set. MetaWorld Hammer, Basketball, and Drawer-Open serve as the non-regression set. The metrics are final success rate and, for Ant, forward speed. We also measure escape probability: the fraction of detected stagnation events followed by a new elite in a different behavior cluster within the window. Two further metrics track policy collapse: the drop in elite-skill success right after a warm-started mutation, and the steps needed to recover. Behavior-cluster entropy over generations completes the set.

The first decisive check runs on the trap-depth family. We plot escape probability against wall depth for elitist anti-goal mutation, tournament ordinary mutation, full method, and matched-compute restarts. The theory predicts that elitist variants fall off sharply with depth while the full method degrades slowly. If restarts track the full method, the attractor hypothesis is refuted for these tasks.

The second decisive check uses FrankaCabinet. The full method should beat R* in our own runs at matched budget, and the gain should vanish under the isotropic-macro-jump ablation. That pattern would show the benefit comes from describing the trap, not from adding entropy.

The third decisive check concerns the escape-versus-collapse tension. With the closeness loss on, we expect lower collapse but also lower escape probability. If both are low with the loss off, the warm start alone is not protecting motor skills, and the escape window needs per-module relabeling instead.

All runs report seeds, LLM calls, and environment steps, so the comparison against restarts can be audited.