When a language model is used as an automatic grader (an "LLM judge" that scores or compares the outputs of other models), it is known to be swayed by irrelevant features. Long answers, fabricated citations, an "EXPERT" label, or a prompt-injected template can all change its verdicts. The field measures these effects almost entirely as verdict flip rates. A flip rate, however, mixes two very different failures. Psychophysics has separated them for decades. In the first, the judge still sees quality differences clearly but has moved its threshold toward the cued answer: a criterion shift. In the second, the cue makes the judge worse at telling good answers from bad ones: a sensitivity loss. The two have opposite practical consequences. A criterion shift can be corrected after the fact by subtracting an offset. A sensitivity loss cannot be fixed this way, because the information the judge needed was never used.
We propose to measure LLM judges the way vision scientists measure observers. We build answer sets whose true quality is varied in controlled, known steps, and inject each bias cue at every step. We then fit how the judge's choice probability rises with the quality gap. A cue that slides this curve sideways is a correctable bias. A cue that flattens it is a blindness that no post-hoc reweighting will remove. Because the quality axis is set by construction, the method never needs the judge's internal likelihood of evidence, which is intractable. It only needs the rank order of a scalar decision signal, and a nonparametric receiver operating characteristic (ROC) analysis supplies that. The output is a per-cue, per-judge diagnosis that tells practitioners which debiasing tools can work and which cannot.
Core Claim
Psychophysics has a standard result that applies directly here. Sensitivity and criterion cannot be separated from a single stimulus intensity. Both Between-subject variability in the influence of mental imagery on conscious perception and the superior colliculus work in Does the Superior Colliculus Control Perceptual Sensitivity or Choice Bias during Attention? show that an apparent change in performance only becomes interpretable once the stimulus strength is varied.
Current LLM-judge bias studies probe exactly one intensity. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge does this with its robustness rate under 12 perturbation types. The Silent Judge does the same with verdict shift rates, up to +30% for GPT-4o under recency cues on ELI5. Both apply a cue to a fixed answer pair and count flips.
Our claim is that adding a controlled quality-gap axis makes the two failures identifiable. For a pairwise judgment between answers $A$ and $B$, write the probability of choosing $A$ as
$$ P(\text{choose } A) = F\big(\beta_c \,\Delta q + \alpha_c\big), $$
where $\Delta q$ is the known quality difference between $A$ and $B$ and $F$ is an unknown increasing link function. The subscript $c$ marks the cue condition (none, cue on $A$, cue on $B$). The slope $\beta_c$ is the judge's sensitivity and the offset $\alpha_c$ is its criterion.
The substantive hypothesis has two parts. Different cues fall into different classes. And the class predicts which correction works. Concretely, length padding and provenance labels should act mostly through $\alpha_c$. Delimiter-hijacking templates of the kind in Cheating Automatic LLM Benchmarks should drive $\beta_c$ toward zero, meaning the judge stops reading content. That collapse should be larger for bigger judges, in line with that paper's finding that Llama-3-70B is more hijackable than 8B. Fabricated citations are the open case.
Where This Sits in Prior Work
Three groups of prior work motivate the proposal by documenting biases as flip rates without decomposing them. CALM (Justice or Prejudice?) injects content-preserving perturbations and reports robustness. The Silent Judge adds the finding that rationales never mention the cue (0% acknowledgment). That finding matters for us because it rules out reading the judge's explanation to diagnose the mechanism. Behavioral psychometrics is the remaining route. The null-model paper shows the most extreme cue, reaching an 86.5% length-controlled win rate on AlpacaEval 2.0 without answering anything.
Two papers supply correction methods that we treat as criterion-only models. Length-Controlled AlpacaEval fits a generalized linear model and zeroes out the length term, which is precisely an additive $\alpha$ correction. TrustJudge replaces integer scores with expected values over score-token probabilities. That gives us a continuous decision variable to use, but it does not ask whether a cue changed sensitivity. Our framework predicts when each of these methods will succeed.
The closest methodological neighbor is Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory. It fits a graded response model to Likert judgments and reports discrimination breadth and reliability. That model has slope-like and threshold-like parameters. However, it estimates quality as a latent variable, covers only point-scale ratings, and treats prompt perturbations as generic noise. We differ in three ways. Quality is set by construction rather than inferred. Judgments are pairwise. Cues are modeled as conditions that may separately move slope and offset. Fixing quality by construction is what makes the decomposition identifiable without assuming the human or LLM latent scale is uncorrupted.
The decision-variable idea comes from Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations. That paper projects network activations onto a discriminant axis to get an unthresholded, criterion-invariant evidence signal. We use the judge's log-odds of preferring $A$ as the analogous signal, and optionally a linear probe on its hidden states.
The intractable-likelihood literature constrains and shapes our estimation choices. Nonparametric inference of prior probabilities from Bayes-optimal behavior recovers biases only if the observer's sensory likelihood is known, and misspecifying that likelihood misattributes noise to bias. For an LLM judge this likelihood is unavailable, so we do not try to model it. Instead we follow the logic of Assessing Generative Models via Precision and Recall, which replaces an intractable density with a curve of tradeoffs computed from samples. Our analog is the ROC curve of the judge's decision variable at each quality gap. Its area is invariant to any monotone transformation and any offset of that variable, so the sensitivity estimate needs no Gaussian or equal-variance assumption.
Research Program
Building the quality ladders. The first component is a set of quality ladders with known steps. For verifiable domains we take correct solutions from the CALM suite's GSM8K and MATH items and create degraded versions with $k = 0$ to $4$ injected errors. Errors are either wrong intermediate steps propagated to the answer or unsupported claims, and each step is checked automatically.
For open-ended domains we use MT-Bench questions and ELI5. We take a strong answer and produce degraded versions by removing or corrupting $k$ distinct informative claims. Human raters then verify on a subsample that quality decreases monotonically in $k$. This also controls for the information-mass confound raised by Explaining Length Bias in LLM-Based Preference Evaluations: degraded versions are length-matched by replacing removed content with neutral filler. Length therefore becomes a separately injected cue rather than a hidden covariate of quality.
Crossing ladders with cues. Every ladder pair is crossed with cue conditions. The cues are length padding, fabricated citations, EXPERT and recency provenance tags (following the Silent Judge cue set), and a null-model-style delimiter template. Each cue is attached to the better answer, the worse answer, or neither, and positions are counterbalanced.
Choosing the judges. Judges are Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct (white-box, matching TrustJudge's setting), Qwen2.5 at two sizes, and GPT-4o as a black-box judge.
Extracting a decision signal. For each judgment we extract a scalar decision variable. For white-box judges this is the log-odds between the "A" and "B" verdict tokens with order-swapping averaged. As an ablation we also use a linear probe on final-layer states trained only on cue-free pairs. For black-box judges the signal comes from a six-level confidence rating, the standard rating-scale ROC procedure from psychophysics, or from the empirical choice frequency over repeated samples at nonzero temperature.
Estimation. From these we estimate, per cue and per quality gap, a nonparametric sensitivity: the ROC area separating the better-on-left from better-on-right trials within that cue condition. We also estimate a criterion: the shift of the choice-probability midpoint along $\Delta q$. As a parametric check we fit the equation above with a probit link.
From diagnosis to correction. A cue is classified as criterion-type if its sensitivity confidence interval overlaps the clean condition while its offset does not. It is sensitivity-type if its sensitivity drops reliably. We then test the practical claim. A per-cue offset correction is fitted on 10% of the ladder items, using cue presence detected by simple string or length features. On held-out items it should restore clean-condition accuracy for criterion-type cues and fail for sensitivity-type cues. For sensitivity-type cues the only remedies are protocol changes, such as stripping delimiters, rewriting answers into a canonical form before judging, or switching judges. The diagnosis tells you to stop trying regression fixes.
Baselines
Naive flip rate. The naive reference point is the current practice: the verdict shift rate of CALM and the Silent Judge, computed on the same pairs at a single quality gap. It shows how often a single-intensity flip rate assigns the same number to cues that our analysis separates into different classes. If flip rate alone already predicts which cues are correctable, the proposal adds nothing. That outcome would count as a failure.
Multi-covariate regression correction. The strongest correction competitor is a generalized linear model in the style of Length-Controlled AlpacaEval, extended with indicator covariates for every cue. It is fitted on the same 10% calibration split and evaluated counterfactually with cues zeroed. This is the strongest competitor because it is cheap, black-box, and already credited with raising Spearman correlation with Chatbot Arena from 0.94 to 0.98 for length. On our ladders it should match our offset correction for criterion-type cues, since it is the same model. The question is whether our diagnosis correctly predicts in advance where it will fail. Beating it means two things. On sensitivity-type cues, the diagnosis-guided protocol change must recover significantly more held-out ladder accuracy than the GLM. And the per-cue predicted recoverability must correlate with the GLM's actual recovery across all cue-judge combinations.
Continuous-score judge. The strongest existing judge-side method is TrustJudge, run with its distribution-sensitive scoring and likelihood-aware aggregation on the identical Llama-3.1-70B judge and prompts. On MT-Bench and Arena-Hard it reported reducing score-comparison inconsistency from 23.32% to 14.89% and pairwise transitivity inconsistency from 15.22% to 4.40%. We reproduce those consistency metrics on our pairs and add cue-condition ladder accuracy. We expect TrustJudge to improve consistency in clean conditions but not to restore sensitivity under hijacking cues. We also test whether its perplexity-weighted tie-breaking amplifies provenance bias, as the Silent Judge result suggests it could.
Latent-quality item response model. The psychometric competitor is the graded response model of Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory. We adapt it to pairwise data, give it cue-dependent discrimination parameters, and let it infer quality latently from judge outputs alone, without access to $k$.
Ablations. The ablation that isolates our key ingredient is the identical slope-and-offset analysis run with the ladder labels removed, that is, with a single intensity. If that version separates cue classes as well as the full analysis, then controlled intensity is not doing the work. A second ablation replaces the nonparametric ROC area with an equal-variance Gaussian sensitivity estimate. It checks whether the assumption-free estimator matters.
Why not just…?
Why not just fit a regression with a covariate for every cue? Because an additive covariate can only represent a criterion shift. If a cue flattens the psychometric curve, the regression will report a moderate average effect and "correct" it. The corrected verdicts on cued pairs will still be near chance, because the content signal was never used. That is plausibly what happens with null-model templates, where perplexity filtering and paraphrasing already failed per Cheating Automatic LLM Benchmarks. This alternative is a real threat for any cue that turns out to be criterion-type. For those cues we expect it to do just as well, and we will say so. The experiment that settles the matter is the held-out ladder accuracy split by diagnosed cue class.
Why not just use item response theory, which already has discrimination and threshold parameters? It is the closest alternative, and we include it as a baseline. Its weakness here is identifiability. When quality is latent and inferred from the same judge whose bias is under study, a cue that compresses the judge's quality scale is hard to tell apart from items that are genuinely closer in quality. The IRT paper also assumes human labels give an uncorrupted latent scale, which our design avoids. The ablation comparing IRT inferred without the ladder labels against our ladder-anchored fit will show whether latent estimation recovers the same cue classification. If it does, the cheaper latent version should be preferred, and we will report it.
Why not just average over order swaps and use continuous scores? Order swapping removes position bias only. The Silent Judge shows provenance and recency effects survive any presentation order. Continuous scores, as in TrustJudge, sharpen the decision variable but do not ask whether a cue changed what that variable tracks. We use continuous scores as our decision variable, so this is a component of the method rather than a rival to it.
Proposed Experiments and Decisive Checks
The first decisive check is whether cue classes exist at all. Across the five cues and five judges, we look for at least one cue whose sensitivity drops beyond its confidence interval while another cue shifts only the criterion. This must hold on both GSM8K-derived and ELI5-derived ladders.
The second check is predictive. Each cue-judge combination gets a diagnosed recoverability (the ratio of cued to clean sensitivity). This should predict the held-out accuracy that the regression baseline actually recovers, with a pre-registered rank correlation threshold.
The third check tests the model-size paradox mechanistically. We predict that the 70B judge's sensitivity under delimiter templates falls further than the 8B judge's, while their clean sensitivities are ordered the other way. This would explain the null-model finding as a blindness rather than a bias.
The fourth check concerns external validity. We use MT-Bench expert pairs from Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, where GPT-4 agrees with experts at 80–85%, with cues injected. Per-cue corrections guided by the diagnosis should yield higher human agreement than both the regression baseline and TrustJudge at equal calibration budget. For AlpacaEval model rankings, cue-aware corrected win rates should not fall below the 0.98 Chatbot Arena Spearman correlation reported for length control.
For fabricated citations, no reported result on Lacuna separates sensitivity from criterion. The citation-condition outcome is therefore a genuine open measurement rather than a replication.
References
- Between-subject variability in the influence of mental imagery on conscious perception
- Does the Superior Colliculus Control Perceptual Sensitivity or Choice Bias during Attention? Evidence from a Multialternative Decision Framework
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
- Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
- TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
- Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
- Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations
- Nonparametric inference of prior probabilities from Bayes-optimal behavior
- Assessing Generative Models via Precision and Recall
- Explaining Length Bias in LLM-Based Preference Evaluations
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena