When several automated judges (language models asked to grade outputs) score the same responses, their errors are not independent. They share biases: they reward length, polished formatting, or confident tone. Averaging their scores then reinforces those biases. The best current fix, CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation, models each judge's score as a mix of true quality and a few shared bias factors. It then uses tensor decomposition, which factorizes a three-way array of score statistics, to separate them. This works because a three-way factorization, unlike a matrix factorization, is unique up to reordering and rescaling of its components. One problem remains. The decomposition returns several components and does not say which one is quality. CARE picks quality as the factor with the strongest shared variation among the most competent judges. When a bias is stronger than the quality signal, that rule selects the bias.
We propose to label the components using a small pilot set of control pairs. Each pair is two versions of the same response that differ only in a suspected bias, for example a verbose and a terse rewrite with identical content. Quality is the same within a pair, so the score differences across judges contain only bias. They reveal the bias directions in judge space without any human labels. Quality is then the decomposition component that lies outside the span of those directions. The idea is borrowed from how particle physicists handle residual systematic errors: they build differential or ratio measurements that cancel the signal and expose the systematic. Here the cancellation is used to anchor an identifiable tensor factorization, not to estimate a background.
Core Claim
Let $s_{ij}$ be judge $j$'s score on item $i$. We model
$$ s_{ij} = a_j q_i + \sum_{k=1}^{K} b_{jk} c_{ik} + \varepsilon_{ij}, $$
where $q_i$ is latent quality, $c_{ik}$ are item-level confounder values, $a_j$ and $b_{jk}$ are judge loadings, and $\varepsilon_{ij}$ is independent noise. For a control pair $(i, i')$ with $q_i = q_{i'}$, the difference vector $d_j = s_{ij} - s_{i'j}$ equals $\sum_k b_{jk}(c_{ik} - c_{i'k})$ plus noise.
The difference vectors therefore span the confounder loading subspace $\mathrm{span}(B)$, and quality cancels exactly. CARE-style third-order moment decomposition recovers the loadings $\{a, b_1, \dots, b_K\}$ uniquely up to permutation and scale. Our labeling rule picks the recovered loading with the largest component orthogonal to $\widehat{\mathrm{span}}(B)$. This rule requires only $a \notin \mathrm{span}(B)$ and places no condition on relative variance. It succeeds exactly where CARE's dominance rule fails.
The claim is testable. As the ratio of confounder variance to quality variance grows past one, CARE's selection should flip to the confounder, while the anchored rule stays correct. When $a \in \mathrm{span}(B)$, no method using these data can separate quality from the confounder. We will measure how performance degrades as $a$ approaches that span.
Relation to Prior Work
CARE supplies the core method: sparse-plus-low-rank and moment-tensor decomposition over conditionally independent judge views. It also motivates the proposal. Its own symmetry-breaking assumption is the weakness the research file identifies as its main open issue. We keep its decomposition and replace only the labeling step.
Two lines of work bear on the identifiability side. Efficient Structure-preserving Support Tensor Train Machine fixes sign and gauge ambiguities in tensor-train factorizations. Numerical CP decomposition of some difficult tensors documents the ill-conditioning of CP fits. Both address ambiguity inside a factorization. Neither addresses which uniquely identified component carries the semantic label "quality." That labeling problem is the one we target.
The Many-Facet Rasch Model approach in Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach is a competitor. It treats rater severity as an additive offset per rater, so it cannot represent a bias whose strength varies with item content, such as a verbosity preference. That paper also motivates our framing of the control-pair set as a pilot calibration done before an evaluator ensemble is deployed.
The physics side constrains the design. Unsupervised in-distribution anomaly detection of new physics through conditional density estimation builds a data-driven null from mass sidebands. The file also notes the risk that fully data-driven background models absorb the signal. Our control pairs are the analogue of a sideband, and they can fail the same way. If a rewrite changes quality, quality leaks into $\widehat{\mathrm{span}}(B)$. We test this leak directly.
Scalability enters as a constraint rather than the core contribution. The third-order moment tensor over $J$ judges has $J^3$ entries. That is trivial for 10 to 50 judges. For larger ensembles we will use sketched or randomized CP (Randomized CP tensor decomposition), so the anchoring step adds almost nothing to the cost of a CARE-sized fit.
Research Program
The work has four parts: build the anchored estimator, derive its identifiability conditions, stress-test it under controlled bias strength, and quantify its failure modes.
Estimator. We first build control pairs. For each targeted confounder (verbosity, formatting, authority cues, and the "beauty" and "authority" biases from the semi-synthetic judgment-bias dataset), a rewriting model produces content-preserving variants of a few hundred responses. Every judge in the ensemble scores both members of each pair. We then estimate $\widehat{\mathrm{span}}(B)$ in one of two ways: principal components of the difference vectors, or, when there are several confounders, a CP decomposition of the difference third moment, which recovers individual confounder loadings. We run CARE's decomposition unchanged on the main evaluation data. Finally, we label quality as the recovered loading with the largest residual after projection onto $\widehat{\mathrm{span}}(B)$, and read item quality from the corresponding item factor.
Analysis. We will prove that the labeling rule is consistent whenever the angle between $a$ and $\mathrm{span}(B)$ is bounded away from zero. We will also derive how many control pairs are needed as a function of that angle and the judges' noise level. This is a standard perturbation argument for estimated subspaces.
Leak analysis. Suppose a fraction of pairs changes quality by an amount $\delta$. We will characterize how far $\widehat{\mathrm{span}}(B)$ tilts toward $a$, and give a diagnostic based on the distribution of pair-difference norms that flags contaminated pairs.
Coverage limit. The method only handles confounders the practitioner thought to probe. We will measure the residual error that an unprobed confounder causes.
Baselines
The naive reference point is the per-item mean of judge scores for continuous tasks and majority vote for preferences. It uses the same judges and the same scoring calls, with the control-pair calls simply unused. Any method that fails to beat this has no case.
The strongest existing competitor is CARE itself. We use CARE-SVD for continuous scores and CARE-Tensor for preferences, on the same judges, data, view partitions and rank choices as our method. We will reproduce its reported numbers before modifying anything. CARE reports up to a 26.8% reduction in mean absolute error relative to majority voting on UltraFeedback, and a 13.4% relative accuracy improvement over MACE and Dawid-Skene on the Summarize preference task (CARE). In the natural regime, where bias is weaker than quality, we do not expect a large gain. Beating CARE there means matching it within noise. In the dominant-confounder regime, beating CARE means lower MAE with non-overlapping bootstrap intervals, together with a correct quality-component selection rate above 90% where CARE's falls below 50%.
The Many-Facet Rasch Model is run with its standard estimation on the same score arrays. It is the principled psychometric alternative and the reference for additive rater corrections.
A supervised stacking baseline fits judge weights by regression on $n$ human-gold-labeled items, with $n$ matched to what a practitioner could afford for a pilot, from 50 to 500.
Two ablations isolate the ingredients. The first is "CARE plus gold labeling": CARE's decomposition, with the quality component chosen as the one best correlated with the same $n$ gold labels. It isolates whether label-free control pairs can replace human labels for anchoring. The second is "projection-out": remove $\widehat{\mathrm{span}}(B)$ from the raw judge-score matrix and average, with no tensor decomposition. It isolates whether the unique, non-orthogonal factorization matters. All methods share the same judge calls and hyperparameter search budget. Control-pair judge calls are also charged to the scoring budget of the naive and CARE baselines, which may spend them on extra evaluation items instead.
Why not just…?
Why not just label CARE's components with a few gold-labeled items? This is a real threat. If a few dozen gold labels reliably pick the right component, control pairs add little. The weakness is that gold labels cost human effort for every new domain and rubric. In addition, when confounder and quality are correlated in a small labeled sample, correlation-based selection can pick the wrong component. The "CARE plus gold labeling" ablation settles this. We report selection accuracy and MAE as functions of $n$ and of the correlation between quality and confounder in the labeled items. If gold labeling matches our method at $n \le 50$ across the variance-ratio sweep, we will report that honestly and reposition control pairs as a label-free alternative.
Why not just project the bias direction out of the score matrix and average? This is the simplest use of control pairs and needs no tensors. It works only if the quality loading $a$ is orthogonal to $\mathrm{span}(B)$. Real judges that are good at assessing quality are often also the ones that respond to polish, so $a$ and $B$ will overlap. Projecting then deletes part of quality along with the bias. The tensor decomposition returns oblique factors, which keeps the quality component intact. We will sweep the cosine between $a$ and $B$ in semi-synthetic data. The prediction is that projection-out matches our method near orthogonality and degrades steadily as the overlap grows. If the gap is negligible at realistic overlaps, estimated from real judge ensembles, the tensor step is unnecessary, and we will say so.
Why not just correct each judge for measured features such as response length? Regressing scores on observed length handles one measured confounder. It cannot handle confounders without a clean feature, such as authority tone or "beauty" cues. Control pairs estimate judge sensitivity to a perturbation without requiring a scalar feature for every item. We include a length-regression correction on verbosity alone as a sanity check. There it should roughly tie with our method.
Why not just use MFRM? MFRM's severity term is a constant offset per rater. The CARE results and our model both describe biases whose effect scales with item properties, and an additive offset cannot represent that. Running MFRM on the same data measures this cost directly.
Proposed Experiments
Continuous-score benchmarks. These are UltraFeedback, Summarize, FeedbackQA and Yelp, as used by CARE. We use an ensemble of 10 to 20 open judges, with MAE against human scores and Kendall rank correlation of system rankings as metrics.
Preference benchmarks. These are Summarize comparisons, Chatbot-Arena and SHP, with accuracy against human preferences as the metric.
Decisive check. The semi-synthetic Humans-LLMs Judgment Bias Dataset lets us inject the beauty and authority biases at a controlled strength. We sweep the ratio of confounder variance to quality variance from 0.1 to 10. We also inject the bias preferentially into the judges that are most competent on clean data, which is the condition that defeats the dominance rule. For every method we report quality-component selection accuracy and MAE across the sweep. The central prediction is that CARE's selection accuracy collapses near a ratio of one while the anchored rule stays flat. If the anchored rule also collapses, the proposal fails.
Leak test. We deliberately contaminate 0% to 30% of control pairs with quality-altering rewrites, such as factual errors introduced during rewriting. We then measure the resulting tilt of $\widehat{\mathrm{span}}(B)$ and whether the pair-norm diagnostic flags the contaminated pairs.
Coverage test. We inject a second bias for which no control pairs exist and report how much of CARE's error the method still removes.
Scalability check. We scale the ensemble to about 100 judges with randomized CP. We confirm that the anchoring step's runtime is negligible and that selection accuracy is unchanged relative to exact CP.
Out of scope. Transfer to detector systematics, with LHC Olympics sidebands serving as the control pairs, is a natural follow-up. It is not claimed here, because no multilinear confounder-isolation method in the file has been tested on collider data.
References
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
- Efficient Structure-preserving Support Tensor Train Machine
- Numerical CP Decomposition of Some Difficult Tensors
- Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach
- Unsupervised in-distribution anomaly detection of new physics through conditional density estimation
- Randomized CP Tensor Decomposition