Testing Whether LLM Judges Favor Their Own Kin

Open-ended LLM leaderboards such as AlpacaEval 2.0 and Arena-Hard rank models by asking another LLM, the "judge," which of two answers is better. A newer form of contamination threatens these rankings. A candidate model fine-tuned on text written by the judge's own model family can inherit that family's phrasing, and the judge then rewards it for the phrasing rather than for the content. Prior work has shown this effect is large and that the judge cannot consciously recognize the related candidate. What is missing is a way to audit a given leaderboard result and say, with a controlled false-alarm rate, whether the judge's scores depend on candidate lineage beyond what answer quality explains. That question is a conditional independence statement: judge score is independent of lineage given quality. Statistics already has a sharp theory of when such statements can and cannot be tested.

We propose to turn preference-leakage auditing into a conditional independence test with stated assumptions. The test uses the Generalised Covariance Measure, which compares leftover errors from two flexible prediction models. The conditioning variable, "quality," is built from signals that do not come from the judge's family: scores from a panel of unrelated judges, reference-answer agreement, length, and prompt features. The theory tells us exactly which assumption the audit rests on, namely that judge score and lineage can each be predicted reasonably well from these quality signals. We make that assumption visible by reporting the prediction errors next to every p-value.

Core Claim

The Hardness of Conditional Independence Testing and the Generalised Covariance Measure proves that, when the conditioning variable is continuous, no test can both control false positives and have power against every alternative without restricting the data distribution. Its Generalised Covariance Measure (GCM) gets around this by relying on regression accuracy. Every contamination or leakage audit that "controls for ability" is implicitly a conditional independence test and so falls under this result. Existing audits hide the regression assumption inside a curve fit or a calibration step. Our claim is that a cross-fitted GCM test, using non-judge-family quality signals as the conditioning variable, can detect identity- and inheritance-type preference leakage at the sample sizes of a single benchmark. It can do so while keeping its false-alarm rate at the nominal level on lineage-unrelated judge pairs. We also expect the simpler linear-adjustment version of the same audit to fail calibration in a predictable way, because judge scores are bounded and saturate.

The roles of the three directions are uneven. Conditional independence theory supplies the method and its limits. LLM-as-a-judge supplies the application. Benchmark contamination, specifically preference leakage, supplies the hypothesis being tested.

Relation to Prior Work

Preference Leakage: A Contamination Problem in LLM-as-a-judge motivates the work most directly. It trains students on 30,000 UltraFeedback prompts answered by different teachers and defines a Preference Leakage Score: the win-rate inflation when the judge is related to the student's teacher. It reports scores up to 37.1% on Arena-Hard. It also mitigates the effect with contextual calibration on human labels. That paper measures inflation by comparing purpose-trained student pairs and gives no hypothesis test. We differ in two ways. We test a null hypothesis with error control. Our test also runs on any population of submitted models with known or suspected lineage, without retraining matched students.

ConStat detects classical benchmark contamination by fitting a difficulty-correction curve on trusted reference models and measuring the performance gap on rephrased benchmarks. It is a conditional-independence argument in disguise, and its known weakness, contaminated reference models, is a failure of the conditioning variable. We make the conditioning explicit and diagnose it, but we target judge-scored open-ended tasks, where ConStat's rephrase-and-score approach and dynamic generation in the style of DyVal do not apply naturally.

Cheating Automatic LLM Benchmarks shows that fixed-string "null models" reach an 86.5% length-controlled win rate on AlpacaEval 2.0 by hijacking the judge's prompt template. It also shows that perplexity filters and template paraphrasing fail as defenses. We use this setting as a sanity case with a second exposure variable. Justice or Prejudice? documents shared judge biases such as verbosity and position. This constrains our design: such biases are common to many judges, so conditioning on panel scores and length removes them, and the test isolates the family-specific excess.

On the statistical side, GHCM supplies the double-robustness logic we rely on: validity needs only the product of the two regressions' errors to shrink faster than $1/n$. The file also flags a known blind spot of these methods: covariance-based tests like GCM and GHCM detect dependence only through conditional means and can miss dependence that shows up only in higher moments. We test this limitation directly rather than assume it away. SpectralCIT is a candidate second-stage test for higher-dimensional conditioning sets. Sequential Kernel-based Conditional Independence Testing via Adaptive Betting shows that tests which assume the distribution of $X$ given $Z$ is known exactly break down when that distribution must instead be estimated; the file reports false rejection rates approaching 100% in that setting. This is our reason for not using knockoff-style tests here.

Research Program

For each prompt $i$ and candidate model $m$, the judge produces a score or pairwise win probability $Y_{im}$. Lineage $X_{im}$ is either binary metadata (whether the candidate's training data came from the judge's family) or a continuous affinity. The affinity is the log-likelihood ratio of the response under an open model from the judge's family versus an open unrelated model, and it is available whenever an open relative exists. The conditioning vector $Z_{im}$ contains scores from $K$ panel judges chosen to be unrelated to every family under test, response length, agreement with the benchmark reference answer where one exists, and a prompt embedding.

We fit flexible regressions $\hat f(Z)\approx \mathbb{E}[Y\mid Z]$ and $\hat g(Z)\approx\mathbb{E}[X\mid Z]$ with cross-fitting, splitting by candidate model so that no model's responses train its own predictions. We then compute residual products $R_{im}=(Y_{im}-\hat f(Z_{im}))(X_{im}-\hat g(Z_{im}))$. Responses from the same model are correlated, so we sum within models before standardizing:

$$T=\frac{\sqrt{M}\,\bar R}{\hat\sigma_R},\qquad \bar R=\frac{1}{M}\sum_{m}\Big(\frac{1}{n_m}\sum_i R_{im}\Big),$$

where $M$ is the number of candidate models, $n_m$ the number of prompts answered by model $m$, and $\hat\sigma_R$ the standard deviation of the per-model averages. Under the null and the GCM rate condition, $T$ is approximately standard normal. That approximation comes from a central limit theorem over models once the regression errors' product vanishes fast enough. When $M$ is small, we instead calibrate within models by aggregating over prompts and report both versions.

Every audit reports the held-out prediction error of $\hat f$ and $\hat g$, so a reader can see whether the assumption that makes the test valid is plausible. A secondary output is a residualized leaderboard: each model's mean of $Y-\hat f(Z)$ beyond its lineage component, with a confidence interval.

Baselines

The naive reference point is the raw judge leaderboard together with the raw Preference Leakage Score difference, read without any test. This shows how often an apparent inflation is within noise.

The strongest existing method for quantifying the effect is the Preference Leakage Score protocol of Preference Leakage. It reports 37.1% for specific generator–judge pairs on Arena-Hard with Qwen-2.5-14B, and around 19.3% to 22.3% when the teacher is only an ancestor of the judge. We run it on exactly the same students, judges, and prompts. Our test beats it if, at matched data, it flags the same leakage pairs with a calibrated p-value and also returns the correct non-rejection on unrelated pairs, which a raw score cannot certify. No reported result found on Lacuna for the Type I error or power of any hypothesis test for preference leakage, so these numbers will be new.

The strongest existing mitigation is contextual calibration on held-out human labels, which the same paper reports reduces Error Bias from 17.8 to 7.3. Our residualized leaderboard competes with it under the same human-label budget, with the labels used only for evaluation in our case. It beats calibration if it reaches comparable Error Bias without spending labels on fitting.

The key ablation is the same audit with linear regressions in place of flexible ones, which is ordinary regression adjustment on panel scores. It is the one most likely to make the flexible machinery look unnecessary. It gets identical splits and tuning budget, and we will report it even if it matches.

Why not just…?

Why not just run a paired style-swap experiment and a t-test? One could rewrite each response into a family-neutral style and compare judge scores before and after. This is a real randomized intervention and is valid where it applies. Preference Leakage itself notes that paraphrasing can damage reasoning content and instruction compliance, so the swap changes quality along with style. It also answers a different question: what a manipulated response would score, rather than whether a published leaderboard is inflated. We include style swaps as a semi-synthetic ground truth and compare conclusions on the same pairs.

Why not just regress judge scores on panel scores and a lineage dummy? This is the linear ablation, and it is the most serious threat. The hardness theorem predicts it fails when the true relation between judge and panel scores is nonlinear. Bounded 1–10 scores and saturating win probabilities make that likely, and the null-model results reaching 9.55/10 on MT-Bench show how hard ceilings bite. If linear adjustment holds nominal error on unrelated pairs as well as the flexible test does, we will say the method reduces to it and recommend the simpler version.

Why not just average a panel of judges from many families? For producing fairer scores, a panel may indeed be enough, and we treat it as an honest alternative for that goal. It does not answer the audit question of whether the official judge's ranking was inflated. Panels also share cross-family biases such as verbosity and authority, documented in Justice or Prejudice?. The test is also how one checks that a panel member is not itself related to a candidate.

Why not just use calibration with human labels? It works, cutting Error Bias from 17.8 to 7.3. However, it needs fresh labels for each shifting distribution, which is the cost that judges were meant to remove. Our experiments compare it at matched label budgets.

Proposed Experiments

The first experiment is a controlled reproduction. We fine-tune small students on UltraFeedback responses from three teacher families and judge them on Arena-Hard and AlpacaEval 2.0 with judges from each family plus unrelated ones. Identity and inheritance pairs serve as positives. Pairs whose judge is unrelated to all teachers, with $X$ defined against a "pseudo-family," serve as nulls. The decisive check is the false-alarm rate across repeated prompt subsamples on null pairs at $\alpha=0.05$, compared with the power on positive pairs. We expect the flexible test to stay near 5% on nulls while the linear ablation drifts upward.

The second experiment is semi-synthetic. We add a known bias term $\delta\cdot X\cdot h(Z)$ to real judge scores. Here $h$ is either constant, which should be detected, or oscillating in panel score with mean zero. Our test is expected to miss the oscillating case, mirroring the Shah–Peters construction and the moment limitation noted for GCM-type tests. Reporting that failure, and testing whether a variance-based residual statistic recovers it, maps exactly where the audit is blind.

The third experiment is a sanity case. Using AlpacaEval 2.0 null-model outputs from Cheating Automatic LLM Benchmarks, we set $X$ to the presence of template-delimiter tokens. The test should reject overwhelmingly, since panel quality scores for fixed strings should be low while the hijacked judge's scores are high.

The fourth experiment is a natural audit of public models with documented distillation lineage, scored by judges of known family. We report p-values, effect sizes, and regression diagnostics. On a subset of Arena-Hard with human preference labels, we compare the residualized leaderboard against contextual calibration on Error Bias.

The proposal is falsified if the flexible test cannot hold nominal error on unrelated pairs. It is also falsified if, once conditioned on unrelated-panel quality, the leakage reported by the Preference Leakage Score disappears, which would mean the inflation is shared judge bias rather than kinship.