Automatic leaderboards such as AlpacaEval 2.0 and MT-Bench use a large language model as the grader, an approach called LLM-as-a-judge. The judge is checked against human preferences once, on the systems that existed at the time, and its scores are then trusted for every system submitted afterwards. Information retrieval met this failure decades ago. A test collection judged by pooling the outputs of one generation of search systems gives unreliable scores to later systems that did not contribute to the pool. We argue LLM judges go stale the same way, and that the staleness belongs to individual systems rather than to the benchmark as a whole. A judge can stay faithful for most submissions while being badly wrong about the few that learned, deliberately or through optimization pressure, to produce what the judge rewards instead of what humans want.
We propose a cheap test that needs no human labels and asks, for one new system, whether the judge's score still tracks the content of that system's answers. We take the system's own outputs and damage their substance in controlled steps while holding fixed the surface features that judges are known to reward: length, formatting, citations, and tone. We then check whether the judge's score falls as the damage grows. If the score barely moves, the judge is grading something other than content for this system, and its leaderboard number should be marked expired. The test turns the abstract idea of a metric's "precarity" into a number computed for each submission. It also tells the benchmark maintainer where to spend a small budget of human audits.
Core Claim
The claim has two parts. First, a judge's error relative to humans is strongly heterogeneous across systems. The largest errors fall on systems whose outputs carry features the judge overweights, such as injected instructions, padding, authoritative formatting, or best-of-n selection against a judge-like reward model, so a single global agreement number hides them. Second, the slope of the judge's score against content degradation, measured on a system's own outputs with style held constant, is a leading indicator of that system-specific error. It should flag gamed or drifted systems before human comparisons do, and do so better than the judge's raw confidence, disagreement among judges, or perplexity filters.
Where This Sits in Prior Work
Several papers motivate the gap. Diaz's account of depth, breadth, and precarity in metric development holds that a metric judged robust in the lab can mislead under shifting deployment conditions, but it offers no way to measure precarity. The NTCIR-15 WWW-3 overview reports that older search runs evaluated against new relevance judgments they did not help pool were ranked unreliably. That is the reusability failure we carry over to judges. Cheating Automatic LLM Benchmarks gives the extreme case. A "null model" that ignores the instruction and emits a fixed injection payload reaches an 86.5% length-controlled win rate on AlpacaEval 2.0 and 9.55/10 on MT-Bench. Larger judges were more obedient to the injection, and perplexity filters and template paraphrasing did not stop it. That paper attacks; it does not detect. Our test is meant to catch such a system without knowing the attack.
Three papers measure judge weaknesses in general rather than for particular systems, and so constrain our design. Justice or Prejudice? injects twelve kinds of non-substantive perturbation, including fake citations and flattering tone, and measures how often judges are swayed. It characterizes the judge. We characterize whether a given system's outputs sit inside the judge's blind spots, and we reuse its perturbation types as the style features to hold fixed. Judging LLM-as-a-Judge shows that rubric text alone predicts pass/fail verdicts with balanced accuracy above 0.8. It also finds that when a candidate answer is flipped from satisfying to violating the rubric, judges change their verdict only 37.7% of the time. That "stickiness" is exactly the content insensitivity we propose to measure, but that paper measures it once per judge on curated flips, not per submitted system. Length-Controlled AlpacaEval fits a regression that removes the effect of length difference and raises Spearman correlation with Chatbot Arena from 0.94 to 0.98. It corrects one known bias globally and cannot see biases it does not model.
Two papers supply methods. LLM as a Meta-Judge validates metrics by having an LLM degrade reference texts across six damage levels, from paraphrase (level 0) to total hallucination (level 5), and correlating metric scores with damage level. It reports meta-correlation up to 0.956 with human validation on Czech QA. We reuse the graded-degradation protocol with three changes: we apply it to each candidate system's own outputs, we need no references, and we add explicit style preservation. Their protocol validates a metric once; ours asks the question separately for every submission. Judge, Retrieve, or Abstain calibrates abstention thresholds with Clopper-Pearson bounds so that the false discovery rate stays below a target. We borrow that calibration to set the expiry threshold. The unit of calibration is a system rather than a factual claim, which is the main statistical difference.
Research Program
For a system $s$ and a sample of its benchmark outputs, we produce degraded versions at levels $k = 0, \dots, 5$. An LLM editor progressively corrupts the claims and steps relevant to the instruction. Automatic checks then reject an edit if it changes length by more than a small tolerance, alters markdown structure, or adds or removes citations and politeness markers. We define the system's content sensitivity as
$$ S_s = \frac{1}{|P|}\sum_{p \in P} \rho\big(k,\; J(p, y_s^{(k)})\big), $$
where $P$ is the sample of prompts, $y_s^{(k)}$ is the system's answer to prompt $p$ degraded to level $k$, $J$ is the judge's score (a pairwise win probability against the benchmark's reference model, or a Likert score), and $\rho$ is Spearman correlation taken with a minus sign so that a healthy judge gives values near 1. A system's score is marked expired when $S_s$ falls below a threshold. The threshold is calibrated on a pool of systems with human ground truth so that, with high probability, the fraction of non-flagged systems whose judge score misses the human-derived score by more than a tolerance $\epsilon$ stays below a chosen level.
We will build three things. The first is the degradation-with-style-lock tool, validated by having humans confirm on a few hundred items that substance drops and style does not. The second is a testbed of systems with known human-relative judge error. It combines public AlpacaEval 2.0 and MT-Bench leaderboard outputs, with Chatbot Arena standing as the human reference, and adds "stress" systems constructed to make the judge go stale: the null-model attack from the cheating paper; length-padded and fake-citation variants of honest systems built with CALM-style perturbations; and best-of-n selections against the UltraFeedback reward model from ULTRAFEEDBACK. That paper reports win rate rising from 76.5% at n=1 to 91.5% at n=16, and we suspect this gain partly reflects shared judge preferences. For best-of-n and the perturbed systems we will collect small human pairwise panels. The third is an analysis of how often judge error is heterogeneous across systems, which tests the premise of the whole proposal. We will also run a rubric-based extension on HEALTHBENCH, where the degradation flips rubric-relevant clinical content.
Baselines
The naive reference is to do nothing: report the judge's score for every system, as AlpacaEval 2.0 and MT-Bench do today, and measure how many systems exceed error tolerance $\epsilon$ and how far the leaderboard's Kendall tau with Arena drops when stress systems are included.
The strongest existing correction is the length-controlled AlpacaEval regression from Length-Controlled AlpacaEval. That paper reports Spearman 0.98 against Chatbot Arena, and the cheating paper shows its 86.5% null-model win rate was obtained under this length-controlled metric. We will run it on exactly the same system pool and judge. Beating it means our flag-and-exclude procedure, or flag-and-human-audit procedure, gives higher Arena rank correlation on the pool with stress systems. On the original honest-system pool it must not fall below the length-controlled value. On honest systems we expect few flags, so matching is the realistic goal there.
A second competitor uses the judge's own uncertainty, in the manner of Judge, Retrieve, or Abstain, and flags a system when its mean verdict entropy is unusual. A third flags systems on which a panel of three judges (GPT-4-class, Claude-3.5-Sonnet, and an open Llama-3-70B judge) disagree. No reported result was found on Lacuna for judge-disagreement detection of gamed systems. A fourth is the perplexity filter, which the cheating paper reports failing against structured injection, though the file gives no numeric detection rate. All detectors receive the same outputs, the same judge-call budget (matched by having the panel and entropy methods score extra samples), and the same calibration systems for threshold setting. Evaluation is on held-out systems.
Two ablations isolate the key ingredients. Removing the style lock tests whether sensitivity measured with uncontrolled degradation, which changes length and format along with content, loses detection power because the judge's style preferences then move scores in the "right" direction. Replacing per-system sensitivity with a single benchmark-wide sensitivity, essentially the meta-judge protocol, tests whether the per-system unit matters.
Why not just…?
Why not just audit every new system with a few dozen human comparisons? This is the real threat. If 30 to 50 pairwise human labels per system estimate judge error well enough, a label-free detector adds little. We will measure detection accuracy as a function of human-label budget and report where the curves cross. The proposal survives in its stronger form only if degradation sensitivity reaches comparable detection at zero labels, or if using it to choose which systems to audit reaches a target error rate with substantially fewer labels than auditing systems uniformly. If neither holds, the honest conclusion is that per-system human audits should be standard leaderboard practice, which would itself be a useful result.
Why not just extend length control to more covariates? A regression can remove biases someone already named, such as length, citation count, or sentiment, and that is a cheap, competitive fix for known exploits. It cannot remove an exploit nobody parameterized, such as the null model's injection, which is why that model scored 86.5% under the length-controlled metric. Our stress set includes both modeled and unmodeled exploits. A multi-covariate regression baseline should win on the former, and we predict our test wins on the latter.
Why not just harden the judge with a stronger model or input sanitization? The file argues against this. Larger judges were more susceptible to injection in the cheating study. Perplexity filtering and paraphrasing failed there. CALM found that general capability does not ensure fair judging. Hardening also does not address quiet drift from optimization pressure, as in best-of-n against a related reward model, where no adversarial text exists to sanitize.
Why not just use judge disagreement? If the gamed features fool all frontier judges similarly, which CALM's finding that fake citations raise credibility across models suggests, a judge panel will agree while being wrong. This is an empirical question, and the disagreement baseline answers it directly.
Proposed Experiments and Decisive Checks
The first check is the premise. On AlpacaEval 2.0 and MT-Bench outputs, we measure whether the gap between judge score and human score varies across systems far more than sampling noise allows. If it does not, there is nothing to detect per system. The second is detection: area under the ROC curve for identifying systems whose judge error exceeds $\epsilon$, on held-out systems, against the entropy, panel, perplexity, and regression baselines. The null model must be flagged; a sensitivity near zero is expected because its payload does not depend on answer content. The third is leaderboard repair: Kendall tau and Spearman against Chatbot Arena after excluding or human-auditing flagged systems, compared with the 0.98 Spearman reported for length-controlled AlpacaEval on its original pool. The fourth is calibration: whether the realized fraction of stale scores among non-flagged held-out systems stays below the target level. We will report how many calibration systems this requires, since systems are scarcer than claims. The fifth is transfer to rubric judging on HEALTHBENCH, where we test whether per-system sensitivity predicts the stickiness that Judging LLM-as-a-Judge measured globally. The idea fails if the style-lock ablation detects as well as the full method, which would mean we measure generic degradation and not judge blind spots, or if a fixed small human audit dominates it at every budget.
References
- The Principles of Depth, Breadth, and Precarity in Developing Metrics for Information Retrieval Evaluation
- Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task
- Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
- Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
- LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
- Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback