TL;DR
AGORA is a new framework for aggregating scores from multiple LLM judges that uses Item-Response Theory to automatically identify and trust the most reliable models. Unlike traditional methods that struggle with small datasets, AGORA remains highly accurate even with just a few dozen examples, making it a robust, label-free tool for evaluating AI systems at scale.
What Problem Does This Paper Address?
Using "LLMs-as-a-judge" has become a standard way to evaluate generative AI, but individual models often harbor systematic biases—like preferring longer answers or favoring their own outputs. The common fix is to use a "panel" of judges and aggregate their scores.
However, current aggregation methods face a "Goldilocks" problem:
- Simple Averaging: Treats every judge equally, meaning one "dumb" or "noisy" model can easily tank the final score.
- Sophisticated Models (e.g., Dawid-Skene): These use "confusion tables" to learn which judges are reliable. While powerful, they require a massive amount of data to estimate their many parameters. In the real world, human-labeled "gold standard" sets are often tiny (sometimes under 100 items), causing these models to fail due to data sparsity.
Methods
The authors propose AGORA (Aggregation Of Reliability-weighted Assessments), which treats judge aggregation as a latent-trait estimation problem.
Instead of a complex confusion matrix, AGORA uses a Graded-Response Model (GRM) from Item-Response Theory (IRT). It models two key things for every judge:
- Discrimination ($\alpha_j$): A "trust" weight that measures how sharply a judge's scores track the underlying quality of an item.
- Thresholds ($\beta_{j,k}$): Parameters that capture a judge's specific "style," such as whether they are generally lenient or severe.
A critical innovation is the use of a regularized horseshoe prior. This mathematical "shrinkage" tool automatically pushes the influence of uninformative or random judges toward zero, effectively "ignoring" them in the final tally without needing any human labels to tell the system who to trust.
Datasets
The researchers evaluated AGORA across three diverse benchmarks:
- SummEval: 1,600 news summaries rated on coherence, consistency, fluency, and relevance.
- Topical-Chat: 360 dialogue responses rated across six dimensions.
- WMT22: A large-scale dataset of 19,725 translated segments with professional quality scores.
Key Insights and Ideas
- Parameter Efficiency is King: Traditional confusion-table models (like Dawid-Skene) require $K^2$ parameters per judge (where $K$ is the number of rating levels). AGORA only needs $K$. This compactness is why AGORA thrives on small datasets where other models starve for data.
- Agreement Dispersion ($\delta$): The authors introduced a label-free statistic that measures how unevenly a pool of judges agrees. They found that the more "messy" or heterogeneous a judge pool is, the more AGORA outperforms simple averaging.
- Trust Weights as Diagnostics: Even though AGORA learns "trust" weights entirely without human labels, these weights correlate strongly with actual human validity. This means you can use AGORA to "audit" your LLM judges and see which ones are actually performing well.
Key Results
- Superior Data Efficiency: On the SummEval benchmark, as the evaluation set shrinks from 400 items down to 50, AGORA maintains its lead while Dawid-Skene’s performance drops significantly.
- Robustness to "Noise": In experiments where random "uninformative" judges were added to the pool, AGORA's accuracy remained stable, while the performance of simple averaging and Dawid-Skene plummeted.
- State-of-the-Art Performance: AGORA achieved the highest rank correlation with human labels across all tested benchmarks, outperforming other popular aggregators like MACE, GLAD, and CARE.
Limitations and Future Work
While AGORA is a significant step forward, the authors note several areas for growth:
- Ordinal Ratings Only: Currently, AGORA is designed for absolute ordinal scores (e.g., 1-5 stars). It would need modification to handle pairwise preferences (e.g., "A is better than B").
- Independent Dimensions: The model currently treats different evaluation dimensions (like "fluency" vs. "relevance") as independent. Jointly modeling these could potentially improve accuracy further.
- Task Breadth: While tested on summarization, dialogue, and translation, the authors suggest that testing on a wider variety of task families is a logical next step.
Summary updated