TL;DR
Evaluating LLM-as-a-judge systems against human labels is highly sensitive to implicit protocol choices, such as how abstentions are handled and how data is aggregated. This paper demonstrates that identical judge verdicts can swing reported accuracy from 0.551 to 0.899 based on these choices alone, and introduces a rigorous reporting checklist to make future LLM-judge evaluations transparent and reproducible.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly deployed as automated judges to grade AI outputs using structured rubrics. To validate these LLM judges, researchers typically calculate how well their verdicts agree with human gold-standard labels using metrics like accuracy or Cohen's kappa ($\kappa$).
However, this paper highlights a major flaw in current evaluation practices: reported agreement numbers depend heavily on hidden, unstandardized assumptions. Decisions about how to handle cases where a judge or human abstains ("Cannot Assess"), how to map judgment scales, and whether to pool data across criteria can drastically alter the final metric. Because these protocol choices are rarely made explicit, it is currently impossible to reliably compare LLM judges across different papers.
Methods
The authors address this problem by porting established statistical principles from psychometrics, econometrics, and corpus annotation into the LLM evaluation space.
- Formalizing the Measurement Protocol: They define an evaluation as a strict four-part protocol $\mathcal{P} = (s, \mathcal{U}, h, a)$, where $s$ is the judgment scale, $\mathcal{U}$ is the selected population, $h$ is the exception-handling rule (for abstentions/invalid outputs), and $a$ is the aggregation rule.
- Mathematical Identity Proofs: They prove that for binary outcomes, five commonly used metrics—Pearson’s $r$, Spearman’s $\rho$, Kendall’s $\tau_b$, the phi coefficient ($\phi$), and the Matthews Correlation Coefficient (MCC)—are mathematically identical.
- Uncertainty Bounding: They introduce Manski worst-case identification bounds to formally quantify the true performance window when abstentions are excluded from evaluation.
- Empirical Re-analysis: They apply these frameworks to re-evaluate raw data from three recently published LLM-judge studies to show how shifting the protocol changes the study's original conclusions.
Datasets
The paper does not introduce new datasets. Instead, it re-analyzes data from three prominent public LLM-judge evaluations:
- Wataoka et al. (2024): A dataset examining LLM self-preference bias containing 2,238 paired comparisons.
- Jung et al. (2025): A judge cascade dataset containing human preferences and judge probabilities for 4,718 test pairs.
- Hashemi et al. (2024) (LLM-Rubric): A benchmark consisting of 223 human-AI dialogues evaluated across 9 distinct rubric criteria (2,007 total decisions).
Key Insights and Ideas
- Metric Redundancy: Reporting Pearson, Spearman, Kendall, Phi, and MCC together on binary judge data is entirely redundant; they are the exact same statistic under the hood.
- The Kappa Paradox: Cohen’s $\kappa$ is highly sensitive to marginal mismatches. On "easy" rubric criteria where almost all submissions meet the requirement, $\kappa$ will naturally plummet toward zero even if the human and the LLM judge have near-perfect raw agreement.
- Abstention Distorts Performance: Simply dropping "Cannot Assess" cases (exclusion) creates an incomplete picture. By applying Manski bounds, the authors show that a judge boasting a "covered accuracy" of 90% on 60% joint coverage actually only guarantees a full-set accuracy of 54%.
- Aggregation Levels Alter Rankings: Micro-averaging (pooling all criteria decisions together) vs. macro-averaging (averaging per-criterion scores) can result in massive discrepancies (e.g., a 4x difference in $\kappa$ on the same data) because micro-averaging allows frequently scored criteria to dominate the metric.
Key Results
- Massive Accuracy Swings: On the Hashemi et al. rubric benchmark, simply altering the protocol choices (verdict extraction method, thresholding, abstention handling, and aggregation level) caused the reported accuracy to swing from 0.551 to 0.899 on the exact same judge verdicts. It also caused $\kappa$ to cross zero, changing the conclusion from "worse than chance" to "positively correlated."
- Cascade Re-analysis: Re-evaluating the Jung et al. judge cascade showed that accuracy shifted from 0.874 under exclusion, to 0.730 under negative recoding, down to 0.534 when treating abstention as a third class. The true full-set accuracy was bounded widely between $[0.534, 0.923]$.
- Standardized Checklist: The authors distilled their findings into a concrete 11-item Reporting Checklist to ensure future agreement claims are fully reconstructible.
Limitations and Future Work
- Binary Constraints: The exact mathematical identities highlighted in the paper strictly apply to binary scales; multi-class, ordinal, or continuous judgment scales do not collapse cleanly into a single statistic.
- Asymptotic Assumptions: The sampling variance equality derived for $\kappa$ and $\phi$ assumes large sample sizes and matched base rates.
- Unilateral Abstention Cases: The analyzed datasets only featured scenarios where either the judge or the human abstained, leaving fully bilateral abstention dynamics in LLM judging open for future empirical study.
Summary updated