Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Delip Rao, Chris Callison-BurcharXiv 2026 · arXiv

TL;DR

Evaluating LLM-as-a-judge systems against human labels is highly sensitive to implicit protocol choices, such as how abstentions are handled and how data is aggregated. This paper demonstrates that identical judge verdicts can swing reported accuracy from 0.551 to 0.899 based on these choices alone, and introduces a rigorous reporting checklist to make future LLM-judge evaluations transparent and reproducible.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly deployed as automated judges to grade AI outputs using structured rubrics. To validate these LLM judges, researchers typically calculate how well their verdicts agree with human gold-standard labels using metrics like accuracy or Cohen's kappa ($\kappa$).

However, this paper highlights a major flaw in current evaluation practices: reported agreement numbers depend heavily on hidden, unstandardized assumptions. Decisions about how to handle cases where a judge or human abstains ("Cannot Assess"), how to map judgment scales, and whether to pool data across criteria can drastically alter the final metric. Because these protocol choices are rarely made explicit, it is currently impossible to reliably compare LLM judges across different papers.

Figure 2: Rubric-based LLM judging. An LLM judge J (the grader in the figure) grades a submission against the criteria of a rubric, each stating a name, a requirement, a weight, and a polarity, and emits one verdict per criterion: MET (M), UNMET (U), or CANNOT_ASSESS (C). Comparing these verdicts with a human annotator’s gold verdicts yields the agreement numbers whose choice, meaning, and reporting this paper examines; the energy-density criterion shown is the running example of Sec. 3.

Methods

The authors address this problem by porting established statistical principles from psychometrics, econometrics, and corpus annotation into the LLM evaluation space.

  1. Formalizing the Measurement Protocol: They define an evaluation as a strict four-part protocol $\mathcal{P} = (s, \mathcal{U}, h, a)$, where $s$ is the judgment scale, $\mathcal{U}$ is the selected population, $h$ is the exception-handling rule (for abstentions/invalid outputs), and $a$ is the aggregation rule.
  2. Mathematical Identity Proofs: They prove that for binary outcomes, five commonly used metrics—Pearson’s $r$, Spearman’s $\rho$, Kendall’s $\tau_b$, the phi coefficient ($\phi$), and the Matthews Correlation Coefficient (MCC)—are mathematically identical.
  3. Uncertainty Bounding: They introduce Manski worst-case identification bounds to formally quantify the true performance window when abstentions are excluded from evaluation.
  4. Empirical Re-analysis: They apply these frameworks to re-evaluate raw data from three recently published LLM-judge studies to show how shifting the protocol changes the study's original conclusions.

Datasets

The paper does not introduce new datasets. Instead, it re-analyzes data from three prominent public LLM-judge evaluations:

  • Wataoka et al. (2024): A dataset examining LLM self-preference bias containing 2,238 paired comparisons.
  • Jung et al. (2025): A judge cascade dataset containing human preferences and judge probabilities for 4,718 test pairs.
  • Hashemi et al. (2024) (LLM-Rubric): A benchmark consisting of 223 human-AI dialogues evaluated across 9 distinct rubric criteria (2,007 total decisions).

Key Insights and Ideas

  • Metric Redundancy: Reporting Pearson, Spearman, Kendall, Phi, and MCC together on binary judge data is entirely redundant; they are the exact same statistic under the hood.
  • The Kappa Paradox: Cohen’s $\kappa$ is highly sensitive to marginal mismatches. On "easy" rubric criteria where almost all submissions meet the requirement, $\kappa$ will naturally plummet toward zero even if the human and the LLM judge have near-perfect raw agreement.
  • Abstention Distorts Performance: Simply dropping "Cannot Assess" cases (exclusion) creates an incomplete picture. By applying Manski bounds, the authors show that a judge boasting a "covered accuracy" of 90% on 60% joint coverage actually only guarantees a full-set accuracy of 54%.
  • Aggregation Levels Alter Rankings: Micro-averaging (pooling all criteria decisions together) vs. macro-averaging (averaging per-criterion scores) can result in massive discrepancies (e.g., a 4x difference in $\kappa$ on the same data) because micro-averaging allows frequently scored criteria to dominate the metric.

Key Results

  • Massive Accuracy Swings: On the Hashemi et al. rubric benchmark, simply altering the protocol choices (verdict extraction method, thresholding, abstention handling, and aggregation level) caused the reported accuracy to swing from 0.551 to 0.899 on the exact same judge verdicts. It also caused $\kappa$ to cross zero, changing the conclusion from "worse than chance" to "positively correlated."
  • Cascade Re-analysis: Re-evaluating the Jung et al. judge cascade showed that accuracy shifted from 0.874 under exclusion, to 0.730 under negative recoding, down to 0.534 when treating abstention as a third class. The true full-set accuracy was bounded widely between $[0.534, 0.923]$.
  • Standardized Checklist: The authors distilled their findings into a concrete 11-item Reporting Checklist to ensure future agreement claims are fully reconstructible.
The proposed 11-item reporting checklist for reconstructible LLM-judge agreement claims.

Limitations and Future Work

  • Binary Constraints: The exact mathematical identities highlighted in the paper strictly apply to binary scales; multi-class, ordinal, or continuous judgment scales do not collapse cleanly into a single statistic.
  • Asymptotic Assumptions: The sampling variance equality derived for $\kappa$ and $\phi$ assumes large sample sizes and matched base rates.
  • Unilateral Abstention Cases: The analyzed datasets only featured scenarios where either the judge or the human abstained, leaving fully bilateral abstention dynamics in LLM judging open for future empirical study.

Summary updated

Abstract

Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria. The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them. We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations. For non-degenerate binary verdicts, Pearson's $r$, Spearman's $ρ$, Kendall's $τ_b$, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names. Cohen's $κ$ differs from them only through a marginal-mismatch factor in $(0,1]$ and shares their asymptotic variance when judge and human assign the positive verdict equally often. Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction. On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from $0.551$ to $0.899$ and carries $κ$ across zero, without altering a single verdict. We distill the analysis into a reporting checklist that makes agreement claims reconstructible and comparable.
BibTeX
@misc{rao2026agreement,
  author = {Delip Rao and Chris Callison-Burch},
  title = {{Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why}},
  year = {2026},
  eprint = {2606.00093},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.00093},
}
Figure