Reward models (networks that score which of two chatbot answers is better) are increasingly trained on labels produced by other language models acting as judges, because human labels are slow and expensive. Those judges are wrong in a particular way. Their mistakes are often confident and repeatable, so asking the same judge several times and checking whether it agrees with itself does not reveal them. This proposal makes two moves. First, the trust placed in each judge-produced label should come from disagreement between different views: different judge models, different scoring criteria, and both orderings of the pair. Second, that disagreement should be converted, using a small human-labelled set, into a calibrated probability that the label is correct. The reward model then trains against that probability instead of a hard 0/1 label.
The obstacle is cost. Collecting many views over large pools of candidate responses requires comparing every response with every other, which is quadratic in pool size and multiplied by the number of views. We borrow the central trick of scalable multi-view clustering. Every view compares each response against a small shared set of reference responses, called anchors. This makes cost linear in pool size, and because the anchors are shared, the views' scores are aligned by construction. The question we test is whether this cross-view, human-calibrated weighting produces reward models that are more accurate and better calibrated than the two simpler alternatives: self-ensembling a single judge, or filtering out low-confidence judgments.
Core Claim
For reward-model training on judge-generated preferences, the useful reliability signal is how far heterogeneous views disagree, after that disagreement is calibrated against humans. A single judge's self-confidence or self-agreement is not enough, because it fails exactly on the confident errors that hurt most. Turning this signal into a soft training target settles the long-standing ambiguity over whether uncertain samples should be down-weighted or emphasized. The data decides: a pair whose label is likely wrong should be softened or flipped, not merely discounted. Comparing against shared anchors keeps the multi-view signal affordable.
Grounding in Prior Work
Several papers motivate the gap. SCOPE queries a judge in both orderings, measures the entropy of the averaged preference, and uses conformal risk control to accept only judgments whose error rate is bounded. It reports an empirical error of 9.7–9.9% at a 10% target. It uses this signal only at evaluation time and needs logit access.
BiG-SURE builds a bipartite graph between low-temperature anchor answers and high-temperature probe answers. This reduces pairwise cost from quadratic to $O(MN)$. However, it explicitly fails on confident hallucinations, where the model repeats the same wrong answer. The broader overconfidence literature explains why: alignment pushes probabilities toward 0 or 1 (Kong et al.), and verbal confidence is unstable under rephrasing (Kapoor et al.). Together these papers say that any uncertainty signal derived from one model consulting itself is suspect.
Other papers constrain the design. On Cost-Effective LLM-as-a-Judge Improvement Techniques shows that ensembling a judge's samples is a strong, cheap fix. It reaches 85.8% accuracy on RewardBench 2 when combined with criteria injection, and makes reference-calibration prompts redundant once $k \ge 3$. Any proposal must beat self-ensembling at matched cost, not merely beat a single call. ROBIN-HOOD shows that judge variance is heterogeneous across items. Adaptive allocation halves the evaluation cost, for example cutting worst-case correctness error from 0.420 to 0.296 for llama-3.1-8b at 50k queries. This suggests our anchor budget should also be allocated adaptively. Noisy but Valid shows that a small human set suffices to estimate a judge's true- and false-positive rates for valid inference. We reuse this logic: a small human set fixes the mapping from disagreement to label correctness.
Some papers supply the method. The multi-view anchor-graph line contributes the idea that comparing $n$ items against $m \ll n$ anchors gives linear cost. This line includes Fast Multi-view Discrete Clustering with Anchor Graphs, the anchor-alignment problem in FMVACC, and Dual-Calibration Multi-View Clustering. In clustering, anchors learned separately per view are misaligned and need explicit correspondence. In our setting we choose the anchors, so all views share identical anchor responses and the alignment problem disappears. We use none of their spectral machinery, only the bipartite cost structure.
HelpSteer2-Preference supplies the loss family. Its Scaled Bradley-Terry loss weights each pair by the human-reported margin, but does not model label reliability. UAPO is the closest training-time competitor. It learns a per-prompt scalar utility anchor that acts as an implicit uncertainty penalty and is robust to 40% flipped labels. Its uncertainty is learned from the policy's own prompt states, not measured from independent judges, and it carries no calibration to humans.
The tension between inverse uncertainty weighting and the up-weighting of ambiguous regions is addressed directly here. We do not pick a sign in advance. We fit the relationship on human labels.
Research Program
The pipeline has three stages.
Collecting views against anchors. For each prompt with a pool of $K$ candidate responses, we select $m$ anchor responses (typically $m=3$). Selection is stratified by an initial cheap judge score so the anchors span the quality range. Each of $V$ views then compares each candidate against each anchor in both orderings. A view is one combination of judge family and criterion, for example correctness judged by model A, helpfulness by model B, or a rubric-injected prompt. Cost per prompt is
$$C = 2\,V\,K\,m \quad \text{versus} \quad V\,K(K-1)$$
for exhaustive both-order pairwise comparison. The anchor version is linear in $K$, so it wins whenever $K > 2m+1$. A direct preference between candidates $a$ and $b$ under view $v$ is inferred by fitting a per-view Bradley-Terry score to its anchor comparisons. Where a direct comparison $a$-versus-$b$ is also available, we record whether it is consistent with the score inferred through the anchors.
Building reliability features and calibrating. For each training pair we compute four quantities: the within-view bidirectional entropy, as in SCOPE; the spread of the preference across views; the fraction of views that agree with the majority; and the anchor-path inconsistency. We then fit a small monotone calibrator, either isotonic or logistic, on a held-out human-labelled set of a few hundred pairs. The calibrator maps these features to $q_i$, the probability that the majority-view label for pair $i$ matches the human label.
Training on soft targets. The reward model $r_\theta$ is trained with
$$\mathcal{L}(\theta) = -\sum_i \Big[q_i \log \sigma\big(r_\theta(x_i,y^w_i)-r_\theta(x_i,y^l_i)\big) + (1-q_i)\log \sigma\big(r_\theta(x_i,y^l_i)-r_\theta(x_i,y^w_i)\big)\Big].$$
Here $x_i$ is the prompt, $y^w_i$ and $y^l_i$ are the judged winner and loser, and $\sigma$ is the logistic function. This loss handles all three cases. A pair with $q_i \approx 0.5$ contributes no net push. A pair with $q_i < 0.5$ is effectively flipped. A confidently correct pair is trained as usual. Down-weighting and up-weighting thus become special cases decided by data.
On the analysis side, we characterize confident judge errors. These are pairs where a judge's own 8-sample ensemble agrees at least 7 of 8 times yet contradicts the human label. We measure how well each reliability signal detects these errors.
Baselines
The naive reference point is a reward model trained on single-call labels from the strongest available judge, with uniform weights and hard targets. Any method that fails to beat it is not worth its added judge calls.
The strongest existing competitor for producing labels is the self-ensembled, criteria-injected judge from On Cost-Effective LLM-as-a-Judge Improvement Techniques, which reports 85.8% accuracy on 1,753 RewardBench 2 examples. We run it with $k$ chosen so that its total judge calls equal our anchor budget $C$. Its majority labels then train the same reward-model backbone, and its agreement fraction is used as the weight. This is the decisive competitor, because it is cheap, strong, and differs from our proposal only in using one view rather than several.
The second competitor is SCOPE-style filtering: drop pairs whose bidirectional entropy exceeds a conformal threshold at $\alpha=0.10$, then train with hard labels on the rest. Its reported controlled error of 9.7–9.9% at $\alpha=0.10$ (SCOPE) describes judgment acceptance, not downstream reward-model quality. For reward models trained on its filtered output we found no reported result on Lacuna.
The third competitor is UAPO applied to the same single-judge labels. It represents learned rather than measured uncertainty. Its reported AlpacaEval 2 numbers (73.5% length-controlled win rate with Gemma-2-9B) concern policy training, so for UAPO we compare on downstream best-of-N quality instead.
We also include Scaled Bradley-Terry on human labels as an upper reference. The ablations hold total judge calls fixed. The first removes cross-view features, leaving within-view entropy and self-agreement fed through the same calibrator. This isolates the multi-view ingredient. The second replaces soft targets with inverse weighting and the third with positive weighting. These isolate the sign question. The fourth replaces anchors with exhaustive pairwise comparisons on small pools, to check that anchoring loses little information.
All reward models share one 8B backbone, one training recipe, and one hyperparameter sweep of equal size. The human calibration set is excluded from training for every method.
For context, published reward models trained on large human data reach 89.0 on RewardBench for ArmoRM-Llama3-8B (RewardBench) and 94.1 for Llama-3.1-Nemotron-70B-Reward (HelpSteer2-Preference). These are ceilings for orientation, not targets. We found no reported result on Lacuna for an 8B reward model trained on judge-relabelled HelpSteer2-Preference. We therefore define beating the matched-budget self-ensemble baseline as the bar. The proposal clears it with a statistically reliable RewardBench 2 accuracy gain of at least 2 points across three seeds, together with lower calibration error against human preferences.
Why not just…?
Why not just ensemble one judge more times and weight by its agreement? This is the real threat. The evidence that self-ensembling captures stochastic noise well is strong (On Cost-Effective LLM-as-a-Judge Improvement Techniques). Our argument rests on the confident-error failure mode documented by BiG-SURE and the overconfidence literature: a judge that is consistently wrong agrees with itself perfectly. The matched-budget comparison and the confident-error detection analysis settle it. If self-agreement detects confident errors about as well as cross-view features (for instance, AUROC within 0.02) and the downstream reward models tie, we drop the multi-view component and report that negative result.
Why not just filter low-confidence pairs with conformal guarantees, as SCOPE does? Filtering keeps confident errors, which pass the threshold by definition. It also throws away pairs whose labels are informative once flipped. SCOPE's guarantee bounds the error among accepted judgments, but says nothing about the systematically wrong subset. The comparison between soft targets and filtering, broken down by $q_i$ bucket, shows whether flipping low-$q$ pairs adds value beyond dropping them.
Why not just scale the loss by margin, or let the model learn its own uncertainty as UAPO does? Margin scaling needs human margins, which judge-labelled data lacks. A learned anchor sees only the policy's own representations, which inherit the same overconfidence. If UAPO matches our method on downstream best-of-N quality without any human calibration set, that is a cheaper solution and we will say so.
Why not just do exhaustive pairwise comparisons and skip anchors? For small pools this is fine. The ablation measures how much label quality anchoring costs at $K=4$, where both are affordable. Anchors only matter for pools of eight or more responses, which is where the linear cost relation above applies.
Proposed Experiments and Decisive Checks
Training data comes from the 6.6k HelpSteer2-Preference training pairs. These are relabelled by judges, with human preferences retained as gold. We expand them to pools of eight responses per prompt to exercise the anchor setting. A 500-pair human subset serves as the calibration set, and a further held-out human set measures reward-model calibration error. We evaluate on RewardBench and RewardBench 2 accuracy, with particular attention to the Ties and Precise Instruction Following subsets of RewardBench 2, where leading models score below 50%. We also measure best-of-N quality, which RewardBench 2 reports correlates with its scores at $r=0.87$.
The first decisive check is the confident-error subset. Cross-view features must detect these errors substantially better than self-agreement, or the core claim fails. The second is the sign test. Soft targets should beat both inverse and positive weighting, and the fitted calibrator should be non-monotone in within-view entropy if both tendencies in the literature are partly right. The third is the calibration-set size curve at 100, 250 and 500 human pairs, which shows how much human effort the method actually requires. The fourth is the matched-budget head-to-head against self-ensembling on RewardBench 2. It is the result the proposal stands or falls on.
References
- SCOPE: Selective Conformal Optimized Pairwise LLM Judging
- BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
- Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
- Large Language Models Must Be Taught to Know What They Don’t Know
- On Cost-Effective LLM-as-a-Judge Improvement Techniques
- LLM-as-a-Judge on a Budget
- Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges
- Fast Multi-view Discrete Clustering with Anchor Graphs
- Align then Fusion: Generalized Large-scale Multi-view Clustering with Anchor Matching Correspondences
- Dual-Calibration Multi-View Clustering via Compact Anchor Learning
- HelpSteer2-Preference: Complementing Ratings with Preferences
- Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
- Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation
- Modeling Aleatoric Uncertainty for Camouflaged Object Detection
- RewardBench: Evaluating Reward Models for Language Modeling
- RewardBench 2: Advancing Reward Model Evaluation