AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

Junhyung AhnarXiv 2026 · arXiv

TL;DR

AGORA is a new framework for aggregating scores from multiple LLM judges that uses Item-Response Theory to automatically identify and trust the most reliable models. Unlike traditional methods that struggle with small datasets, AGORA remains highly accurate even with just a few dozen examples, making it a robust, label-free tool for evaluating AI systems at scale.

What Problem Does This Paper Address?

Using "LLMs-as-a-judge" has become a standard way to evaluate generative AI, but individual models often harbor systematic biases—like preferring longer answers or favoring their own outputs. The common fix is to use a "panel" of judges and aggregate their scores.

However, current aggregation methods face a "Goldilocks" problem:

  1. Simple Averaging: Treats every judge equally, meaning one "dumb" or "noisy" model can easily tank the final score.
  2. Sophisticated Models (e.g., Dawid-Skene): These use "confusion tables" to learn which judges are reliable. While powerful, they require a massive amount of data to estimate their many parameters. In the real world, human-labeled "gold standard" sets are often tiny (sometimes under 100 items), causing these models to fail due to data sparsity.

Methods

The authors propose AGORA (Aggregation Of Reliability-weighted Assessments), which treats judge aggregation as a latent-trait estimation problem.

Instead of a complex confusion matrix, AGORA uses a Graded-Response Model (GRM) from Item-Response Theory (IRT). It models two key things for every judge:

  • Discrimination ($\alpha_j$): A "trust" weight that measures how sharply a judge's scores track the underlying quality of an item.
  • Thresholds ($\beta_{j,k}$): Parameters that capture a judge's specific "style," such as whether they are generally lenient or severe.
The AGORA workflow: (1) Input judge scores, (2) Fit IRT models to learn trust and severity, (3) Aggregate using a weighted posterior, (4) Map back to the original scale.

A critical innovation is the use of a regularized horseshoe prior. This mathematical "shrinkage" tool automatically pushes the influence of uninformative or random judges toward zero, effectively "ignoring" them in the final tally without needing any human labels to tell the system who to trust.

Datasets

The researchers evaluated AGORA across three diverse benchmarks:

  • SummEval: 1,600 news summaries rated on coherence, consistency, fluency, and relevance.
  • Topical-Chat: 360 dialogue responses rated across six dimensions.
  • WMT22: A large-scale dataset of 19,725 translated segments with professional quality scores.

Key Insights and Ideas

  • Parameter Efficiency is King: Traditional confusion-table models (like Dawid-Skene) require $K^2$ parameters per judge (where $K$ is the number of rating levels). AGORA only needs $K$. This compactness is why AGORA thrives on small datasets where other models starve for data.
  • Agreement Dispersion ($\delta$): The authors introduced a label-free statistic that measures how unevenly a pool of judges agrees. They found that the more "messy" or heterogeneous a judge pool is, the more AGORA outperforms simple averaging.
  • Trust Weights as Diagnostics: Even though AGORA learns "trust" weights entirely without human labels, these weights correlate strongly with actual human validity. This means you can use AGORA to "audit" your LLM judges and see which ones are actually performing well.
A comparison of parameterization: Dawid-Skene's confusion matrix becomes sparse and unreliable as data shrinks, while AGORA's latent-trait model remains stable.

Key Results

  • Superior Data Efficiency: On the SummEval benchmark, as the evaluation set shrinks from 400 items down to 50, AGORA maintains its lead while Dawid-Skene’s performance drops significantly.
  • Robustness to "Noise": In experiments where random "uninformative" judges were added to the pool, AGORA's accuracy remained stable, while the performance of simple averaging and Dawid-Skene plummeted.
  • State-of-the-Art Performance: AGORA achieved the highest rank correlation with human labels across all tested benchmarks, outperforming other popular aggregators like MACE, GLAD, and CARE.
Performance vs. evaluation set size (N). AGORA (blue line) consistently stays at the top as N decreases, showing much better data efficiency than baselines.

Limitations and Future Work

While AGORA is a significant step forward, the authors note several areas for growth:

  • Ordinal Ratings Only: Currently, AGORA is designed for absolute ordinal scores (e.g., 1-5 stars). It would need modification to handle pairwise preferences (e.g., "A is better than B").
  • Independent Dimensions: The model currently treats different evaluation dimensions (like "fluency" vs. "relevance") as independent. Jointly modeling these could potentially improve accuracy further.
  • Task Breadth: While tested on summarization, dialogue, and translation, the authors suggest that testing on a wider variety of task families is a logical next step.

Summary updated

Abstract

LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.
BibTeX
@misc{ahn2026agora,
  author = {Junhyung Ahn},
  title = {{AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory}},
  year = {2026},
  eprint = {2611.05556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05556},
}
Figure