Label-Free Fitness for Automated Reward Design via Weighted Judge Voting

Coding language models can now write reward functions for robots. Eureka showed that GPT-4, given a simulator's source code, can evolve reward programs that beat human-written ones. One manual step remains. Every candidate reward is scored by a hand-written success metric, and the best one survives. When success is hard to put in code ("fold the cloth neatly," "walk naturally"), the loop has nothing to select with. The usual fix is to ask a single vision-language model (VLM), a model that answers questions about images or video, whether each rollout succeeded. That judge has blind spots, and an evolutionary search is very good at finding them.

We propose replacing the hand-written success metric with a weighted vote of many heterogeneous judges. Each judge's reliability is estimated without any labels, from the pattern of agreement among the judges. Judges from the same model family are merged before voting so that their shared errors are not counted twice. Judges that turn out to be worse than chance get negative weight. The claim is specific. On benchmarks where a ground-truth success metric exists but is hidden from the search, this label-free, family-corrected vote should pick nearly the same reward programs as the hidden metric would. It should also pick reward programs that exploit a judge's blind spots, which we call hacks, far less often than a single strong VLM judge or an unweighted majority vote.

Core Claim

The fitness question in reward search is "did this policy's rollout succeed?" That is a binary classification problem answered by several imperfect classifiers, and it has no labels. This is the setting that spectral meta-learning was built for. Estimating the accuracies of multiple classifiers without labeled data shows that when classifiers make independent errors, the off-diagonal entries of their vote covariance matrix form a rank-one matrix whose factors reveal each classifier's balanced accuracy. The method then recovers each classifier's sensitivity $\psi_i$ (accuracy on positives) and specificity $\eta_i$ (accuracy on negatives) with no ground truth.

Let $v_i \in \{-1,+1\}$ be judge $i$'s verdict on a rollout. The maximum-likelihood aggregate is then $$\hat y = \mathrm{sign}\Big(\sum_i w_i v_i + c\Big), \qquad w_i = \tfrac12 \log\frac{\psi_i\,\eta_i}{(1-\psi_i)(1-\eta_i)},$$ where $c$ is an offset set by the class prior. The weight $w_i$ is negative exactly when $\psi_i + \eta_i < 1$, that is, when the judge is worse than chance. The rule therefore treats a systematically wrong judge as evidence for the opposite answer, rather than as noise to be averaged away. This mirrors the finding of Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling, which showed that calibrated weights for math answer selection should be negative for low verifier scores. That paper learns its weights from a pre-computed labeled dataset. We estimate ours without labels, and we re-estimate them every generation, because the rollouts being judged shift as the search proceeds.

Why the obvious combinations are not enough

Two findings constrain the design.

The first is that the independence assumption fails in practice. Hidden Clones shows across 17 VLMs from 8 families that models from the same lineage make correlated errors. Helping or Herding? shows that reward-model ensembles sharing a pretrained base agree on the same hacks, such as list-formatted answers and extractive summaries. Applied naively, spectral estimation would read a cluster of clones as one highly reliable judge.

The second is an identifiability limit. Causal Discovery in the Wild proves that recovering expert competence without labels needs at least $2m-1$ conditionally independent experts, where $m$ is the number of states per feature. For binary success ($m=2$), that means at least three independent groups.

Our method therefore works at the level of families. Judges are first grouped by lineage, using known provenance plus clustering on the residual covariance left after a rank-one fit. Each family then contributes one pooled verdict, and spectral estimation runs across families. This requires at least three genuinely distinct families. To add diversity beyond model lineage, we also include judges that see a different input: language-model judges that read numeric state-trajectory summaries instead of video.

The closest bridge papers all aggregate uniformly. Improving RLHF with Efficient Reward Model Ensemble averages LoRA reward heads or subtracts their standard deviation. VELR uses ensemble variance to flag out-of-distribution samples and falls back on a 32-billion-parameter teacher. CoRe selects code rewards with a single Gemini 2.0 Flash judge. Eureka needs a ground-truth fitness function. None of them estimates judge reliability without labels, and none corrects for correlation within a family.

Research Program

We rebuild the Eureka loop with one change: fitness comes from the judge vote. Each generation proceeds in four steps. First, the language model samples a batch of reward programs, and a policy is trained on each in simulation. Second, each trained policy is rolled out $K$ times. Third, every rollout is shown to a judge pool made of VLMs from at least three families, each asked a fixed yes/no success question, plus text judges reading state summaries. Fourth, family-pooled spectral weights are estimated from that generation's verdict matrix alone, and each candidate's fitness is its estimated success rate under the weighted vote. The rest of the Eureka loop, including reflection and mutation, is unchanged. The ground-truth metric is logged but never shown to the search.

The analysis asks three questions. Are the estimated judge accuracies correct? We check them against hidden labels, and we check the $1/\sqrt{n}$ convergence rate the spectral method predicts. Do judge reliabilities actually change across generations as policies become more capable and stranger? Does family pooling prevent the search from converging on hacks that one family of judges misses? A hack is defined as a candidate whose vote-estimated success is high but whose hidden success is low.

Baselines

The naive reference point is an unweighted majority vote of the same judge pool, run as fitness in the same loop. It uses the same judges, rollouts, policy-training budget and number of generations. It isolates whether weighting matters at all. The direction-1 literature, including the plurality-voting failures in Causal Discovery in the Wild, predicts it degrades when weak judges are the majority.

The strongest practical competitor is the single strong VLM judge used by CoRe, which reports 99.0% average success across 7 Meta-World tasks with Gemini 2.0 Flash judging. We run CoRe's judge as the fitness signal in our loop and also run the full CoRe pipeline as its authors configure it. We match total judge API spend by giving the single judge more rollouts per candidate.

The upper reference is Eureka with its hidden ground-truth fitness. On Isaac Gym, Eureka reports beating human rewards on 83% of 29 tasks with a 52% average normalized improvement (Eureka). We will reproduce the oracle loop ourselves under our compute budget rather than compare against the reported figures directly.

A labeled-weighting baseline estimates judge weights from a small human-labeled set of rollouts collected in the first generation, in the style of the validation-weighted systems in the file. We give it the same total number of labels a practitioner might tolerate.

Three ablations isolate the key ingredients. The first uses spectral weights without family pooling. The second uses family pooling with uniform weights. The third keeps the full method but clips weights to be non-negative.

We count as success three outcomes. The method must beat the majority vote and the single-judge baseline on hidden success with matched cost. It must reach at least 90% of the oracle's normalized score. It must have a lower hack rate than every baseline.

Why not just…?

Why not just use one strong VLM as the judge? This is the most serious threat, because CoRe's reported Meta-World numbers are already near ceiling. It is not a strawman. Our argument is that the ceiling hides selection pressure. An evolutionary search optimizes against its judge, and a single judge's perceptual blind spots become targets, which is the herding failure documented for reward models in Helping or Herding?. The experiment settles this through the hack rate and through SoftGym and Isaac Gym dexterity tasks, where success is visually subtle. If a single judge matches us on hidden success and hack rate across those tasks, the ensemble is not worth its cost, and we will report that.

Why not just take a majority vote, or subtract ensemble disagreement? Uniform voting counts a family of clones several times over. Disagreement penalties do not help when judges agree on a hack: Improving RLHF with Efficient Reward Model Ensemble found that subtracting the standard deviation performed about the same as the plain mean. The majority-vote and uniform-weight ablations test this directly.

Why not just label twenty rollouts and fit weights? This may work well, and it is a real threat. The question is whether weights fit on early rollouts go stale as the policies drift. We test this by measuring how much estimated judge accuracies shift between generations. If they barely move, a small labeled set is the simpler answer. In that case the contribution narrows to fully label-free operation plus family correction, and the note will say so.

Decisive Checks

We run on 9 Isaac Gym control tasks plus a subset of the Dexterity tasks, 7 Meta-World tasks and 3 SoftGym tasks, hiding each benchmark's success metric during search. We report four quantities. The first is human-normalized score, as in Eureka. The second is selection agreement with the oracle's top candidate. The third is the error between spectrally estimated and true judge accuracies. The fourth is the hack rate.

Three results would be decisive. The first is spectral accuracy estimates that track the truth only when families are pooled; this confirms that correlation within families is the binding problem. The second is non-trivial negative weights emerging for some judge on some task; this shows that the negative-evidence principle transfers from answer selection to reward selection. The third is a gap in hack rate over the single-judge baseline that widens in later generations. Failure on the first check, with accurate estimates even without pooling, would mean the family correction is unnecessary. Failure on the third would mean the vote adds little over a strong single judge.