Rationing the Judge: Hybrid Evolutionary RL as a Bandwidth Control Against Perceptual Reward Hacking

When an image model is trained to raise a learned quality score, it eventually learns to fool the scorer instead of improving the image. The usual responses are to restrict how much the model can change, as with low-rank adapters, or to stop training early. Both are blunt. This proposal treats the problem as one of information flow. A policy can only exploit an evaluator as far as it can learn the evaluator's blind spots. A gradient passed back through the evaluator reveals a great deal about those blind spots. A single "keep the best checkpoint" decision reveals almost nothing. Hybrid evolutionary reinforcement learning offers a middle setting that can be tuned. In this method, a population of policies is mutated and selected by whole-episode scores, while a gradient-trained learner shares the population's experience and is periodically inserted back into it.

We propose a split-evaluator design. The gradient learner is trained on a cheap differentiable quality model. A second, independent quality judge is used only for evolutionary selection. The amount of information the population can extract from that judge is bounded by the size of the population and how often selection happens. Our central claim is falsifiable. Human-rated quality is an inverted-U function of how much information about the trusted judge reaches the policy. Hybrid evolutionary RL can be tuned to sit near the peak, beating both gradient-through-the-judge (too much information) and checkpoint selection (too little).

Core Claim

The perception literature documents the failure we target. Augmenting Perceptual Super-Resolution via Image Quality Predictors backpropagates through no-reference quality models (MUSIQ, NIMA) to fine-tune Real-ESRGAN and finds high-frequency "adversarial artifacts" that raise the score while degrading the image. Low-rank adaptation is needed to contain them. B-Spar finds the step-level version of the problem in RL photo retouching: dense proxy rewards teach agents to jitter sliders for imperceptible numeric gains. Its fix replaces them with sparse binary feedback, which a Bayesian neural network then smooths. Human Preference Score shows that feature-statistic metrics are biased toward texture and miss structural failures. All three papers motivate the proposal. None of them asks how hard the policy is allowed to push on the evaluator it trusts most.

On the evolutionary side, Evolution-Guided Policy Gradient in Reinforcement Learning and Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning supply the method. They define the loop of population rollouts feeding a shared replay buffer, an off-policy learner trained from that buffer, and the learner being inserted back into the population. Genetic Soft Updates supplies weight-averaging updates that keep insertion from destabilizing the population. These works were evaluated on control, games and placement, where the fitness function is trusted. We use them in a setting where the fitness function is the thing being protected.

We reject one tempting claim at the outset: that evolution is immune to metric hacking because it uses no gradients. Black-box search can still find adversarial inputs, only more slowly. The defensible claim is quantitative. Suppose truncation selection keeps the top $k$ of $N$ individuals every generation for $G$ generations. Then the policy can learn at most

$$I_{\text{sel}} \le G \log_2 \binom{N}{k}$$

bits about the selection judge's ranking. This holds because each selection event is one choice among $\binom{N}{k}$ subsets. Gradient access through the judge has no comparable bound, and one-shot checkpoint selection corresponds to $G=1$. The quantity $I_{\text{sel}}$ is a dial, set by $N$, $k$ and how often selection runs. Our hypothesis is that human preference peaks at an intermediate value. The judge's useful signal arrives early, and its idiosyncratic blind spots need many bits to locate.

Research Program

The main testbed is tool-based photo retouching on MIT-FiveK, the setting of B-Spar. An agent applies a short sequence of exposure, contrast, tone-curve and color operations to a RAW-derived image. Rollouts take milliseconds, the operators need not be differentiable, and the strongest existing RL method can be compared directly.

The gradient learner is an off-policy actor-critic agent. Its reward is the dense per-step change in a differentiable no-reference quality model, which we call evaluator A (MUSIQ, as in Zhang et al.). The population consists of small policy networks of the same architecture. Their fitness is the final-image rating from an independent judge, evaluator B. B is a Q-Insight-style multimodal model trained with rule-based reinforcement learning, as in Q-Insight. We query B only for final images and never differentiate through it. Every individual's transitions go into the shared buffer. The learner is inserted into the population with genetic soft updates. The deployed policy is the population champion according to B.

Because fitness is computed per episode, B never sees the step-level slider jitter that B-Spar identified. Jitter that does not change the final image earns no selection advantage. The work has three parts. First, we build this split-evaluator system. Second, we measure how well exploits transfer across evaluators: we optimize images hard against A and record how often B and humans also rate them highly. The design only makes sense if A's exploits transfer weakly to B. Third, we sweep $I_{\text{sel}}$ by varying population size, truncation ratio and selection frequency, and trace human preference against it. As a secondary check of generality, we repeat the exploit-transfer and bandwidth sweep on Real-ESRGAN super-resolution with a population of low-rank adapters on the HGGT Test-100 setup of Zhang et al.

Baselines

The naive reference points are the unedited input image and expert C's retouch, the conventional MIT-FiveK target. These show whether any learned editing helps under human judgment.

The strongest existing method on this exact task is B-Spar. It is an RL retouching agent built on a 2.01B-parameter multimodal model, with Bayesian smoothing of sparse rewards and prior-guided sampling. On MIT-FiveK it reports LPIPS 1.03, a user-study score of 6.8/10 against 5.3/10 for expert B, and roughly 33.5% relative perceptual improvement over PhotoArt. We will run B-Spar on the same test split if code is available. We will add its outputs to our own pairwise human study rather than compare against its reported 6.8/10, which came from a different protocol. Beating it means a statistically significant pairwise human win rate above 50% at matched inference latency class, plus no loss on LPIPS to expert retouches.

The strongest gradient-side competitor transfers Zhang et al.'s recipe: the same actor-critic agent with evaluator A as reward and a capacity constraint, with checkpoints chosen by B. This is the "minimal bandwidth" end of the dial. We also run the "maximal bandwidth" end: policy-gradient RL with B's score as the terminal reward, in the style of the group-relative optimization Q-Insight uses for its judge.

Pure evolution against B, in the spirit of Evolution Strategies at Scale, tests whether the gradient learner contributes anything.

The key ablation is identical hybrid ERL with A used for both learning and selection. It isolates whether the split, rather than the population, does the work.

All methods get the same environment-step budget, the same number of B queries, and the same hyperparameter search budget of a fixed number of trials tuned on a validation split scored by a third evaluator. The third evaluator is BiRQA, from BiRQA, and it is never used in training.

Why not just…?

Why not just add B to the reward as a regularizer, or ensemble A and B? This is a real threat when B is differentiable, but it then gives the policy gradient access to B, which is the maximal-bandwidth condition. When B is a reasoning multimodal judge, ensembling turns into policy-gradient RL on B. That is our maximal-bandwidth baseline. If it matches ERL on human preference, our inverted-U hypothesis is wrong, and the experiment will show it.

Why not just train on A and pick the checkpoint B likes best? This is the $G=1$ end of the dial, and it is cheap. It loses if B carries useful signal beyond a single choice, for example by steering search toward edits that A undervalues. The bandwidth sweep measures exactly this. If the curve is flat from $G=1$ upward, checkpoint selection is the right answer and the population is unnecessary.

Why not just use B-Spar's Bayesian reward smoothing? B-Spar solves sparse feedback, not evaluator exploitation. Its smoothed landscape is still a target the agent optimizes with high bandwidth. It also depends on a 2B-parameter prior model. We compare directly on its own benchmark. A loss to B-Spar would mean the bandwidth framing adds nothing over better reward modeling in this domain.

Why not just use pure evolution strategies against B? Evolutionary methods throw away step-level transitions and are sample-inefficient, as reviewed in Derivative-free reinforcement learning: a review. Under a matched step budget we expect pure evolution to plateau lower. If it does not, the gradient learner is unnecessary.

Proposed Experiments

Human pairwise preference on the MIT-FiveK test split is the primary metric. Secondary metrics are LPIPS to expert retouches, held-out BiRQA, and the evaluator gap. The evaluator gap is A's score minus the human rating on a normalized scale, and it measures hacking directly.

The first decisive check is exploit transfer. If images in the top 1% of A-scores are rated by B as highly as genuinely good edits, the split design cannot work, and we would report that negative result.

The second check is the bandwidth curve. We predict an interior peak in human preference over $I_{\text{sel}}$, with the evaluator gap rising monotonically. A monotone preference curve in either direction falsifies the claim.

The third check is the same-evaluator ablation. If ERL with A used for both roles matches the split version, the gain comes from population diversity, not from rationing the judge.

For super-resolution, Zhang et al. report about 12% LPIPS-ST improvement over Real-ESRGAN on HGGT. We test whether split-evaluator selection over adapters keeps that improvement while lowering the evaluator gap. For KonIQ-based judge calibration we found no reported result on Lacuna beyond Q-Insight's 0.806 average PLCC, so we will calibrate B on held-out human scores ourselves before trusting it as a selector.