LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generation

Anu Pradhan, Alexandra Ortan, Apurv Verma, Madhavan SeshadriarXiv 2025 · arXiv

TL;DR

Evaluating Retrieval-Augmented Generation (RAG) systems in specialized domains like law is notoriously slow and expensive due to the need for expert human raters. While using Large Language Models (LLMs) as automated judges offers a scalable alternative, highly skewed data distributions can severely distort traditional statistical metrics. This paper provides a robust, mathematically principled framework using Gwet’s AC2 and the Wilcoxon Signed-Rank Test to reliably select LLM judges and compare RAG systems.

What Problem Does This Paper Address?

In highly specialized fields like legal research, the stakes for AI recommendation quality are exceptionally high—a minor hallucination or missed precedent can have serious professional consequences. While human evaluation remains the gold standard, it creates a massive bottleneck. Traditional automated metrics like BLEU or ROUGE fail because they require exact reference texts and cannot evaluate nuanced dimensions like readability, completeness, or legal correctness.

Using an "LLM-as-a-Judge" is a promising alternative, but it introduces a new challenge: how do we know we can trust the judge? AI evaluation datasets often suffer from extreme data skew (e.g., a system might get almost all high scores or almost all low scores on a specific metric). Under these conditions, standard statistical methods for measuring agreement between human and AI judges break down, leading to misleading conclusions.

Methods

The authors from Bloomberg tackle this problem by breaking the evaluation pipeline into two distinct statistical steps:

  1. Evaluating and Selecting the LLM Judge: Instead of relying on a single metric, they propose a multimetric Inter-Rater Reliability (IRR) framework to measure how well various LLMs align with human experts. They compare traditional metrics like Cohen’s Kappa and Krippendorff’s Alpha against rank correlation coefficients (Spearman’s Rank and Kendall’s Tau) and a more recent technique, Gwet’s AC2 (with linear and quadratic weighting).
Comparison of IRR Metrics for Legal RAG System Evaluation
  1. Comparing Competing RAG Systems: To statistically compare two different RAG systems evaluated by an LLM judge, they advocate for the Wilcoxon Signed-Rank Test (WSRT). Because legal evaluation data is rarely normally distributed, this non-parametric test is ideal. To control for false positives across multiple evaluation dimensions (relevance, completeness, correctness, etc.), they apply Benjamini-Hochberg (B-H) corrections.
Table 2: Statistical test comparison for LLM evaluation. Our choice: Wilcoxon for paired A/B testing with magnitude sensitivity.

Datasets

The study was conducted using a realistic industry dataset from Bloomberg Law:

  • 117 anonymized legal user queries curated from practicing attorneys and librarians.
  • Two distinct RAG systems were evaluated:
  • System A: Standard BM25 retrieval paired with an open-source LLM summarizer.
  • System B: An enhanced retrieval system paired with proprietary GPT-4 as the summarizer.

Key Insights and Ideas

  • The Prevalence Paradox: The most striking insight is that traditional metrics like Krippendorff’s Alpha can drastically underestimate judge agreement when data is heavily skewed. If an LLM and a human agree 90% of the time, but 90% of the total data falls into a single score category, the mathematical "chance agreement" term inflates, driving the final reliability score toward zero.
  • Gwet’s AC2 to the Rescue: Gwet’s AC2 avoids this pitfall by changing how chance agreement is calculated, making it highly robust against skewed distributions.
  • The Skewness of Hallucinations: As shown in the paper's skewness analysis, metrics related to hallucinations exhibit extreme right skewness (meaning hallucinations are rare but severe when they happen). This statistical reality completely invalidates standard parametric tests like the paired t-test, mandating non-parametric alternatives like Wilcoxon.
Distribution skewness of evaluation metrics

Key Results

  • GPT-4o Emerged as the Best Judge: Across all tested models (including LLaMA2-70B, Claude Opus3, and Mistral), GPT-4o demonstrated the highest agreement with human experts, achieving a Gwet’s AC2 (quadratic) score of 0.78 and a Spearman correlation of 0.73 on search relevancy.
LLM judge agreement and ranking consistency on Search Relevancy
  • RAG System Trade-offs: Using their statistical framework, the authors found that System B (Advanced Retrieval + GPT-4) was significantly superior in core accuracy metrics: relevance ($p = 0.0358$), completeness ($p = 1.215\times10^{-18}$), and correctness ($p < 0.05$). However, System A actually performed better at readability ($p = 0.01997$) and preventing extrinsic hallucinations ($p = 0.0204$), highlighting a critical trade-off between precision and presentation quality.

Limitations and Future Work

The authors acknowledge a few constraints:

  • Proprietary Constraints: Due to corporate confidentiality and proprietary data agreements at Bloomberg Law, the complete evaluation results and exact system specifications could not be fully shared.
  • Dataset Size: While 117 queries are realistic for expert-curated legal evaluations, larger datasets would further validate the statistical power of these methods.
  • Future Directions: The paper suggests that future work should explore how these robust statistical frameworks apply to other highly specialized domains (like medicine or finance) that face similar data-skew and high-stakes evaluation bottlenecks.

Summary updated

Abstract

The evaluation bottleneck in recommendation systems has become particularly acute with the rise of Generative AI, where traditional metrics fall short of capturing nuanced quality dimensions that matter in specialized domains like legal research. Can we trust Large Language Models to serve as reliable judges of their own kind? This paper investigates LLM-as-a-Judge as a principled approach to evaluating Retrieval-Augmented Generation systems in legal contexts, where the stakes of recommendation quality are exceptionally high. We tackle two fundamental questions that determine practical viability: which inter-rater reliability metrics best capture the alignment between LLM and human assessments, and how do we conduct statistically sound comparisons between competing systems? Through systematic experimentation, we discover that traditional agreement metrics like Krippendorff's alpha can be misleading in the skewed distributions typical of AI system evaluations. Instead, Gwet's AC2 and rank correlation coefficients emerge as more robust indicators for judge selection, while the Wilcoxon Signed-Rank Test with Benjamini-Hochberg corrections provides the statistical rigor needed for reliable system comparisons. Our findings suggest a path toward scalable, cost-effective evaluation that maintains the precision demanded by legal applications, transforming what was once a human-intensive bottleneck into an automated, yet statistically principled, evaluation framework.
BibTeX
@misc{pradhan2025llm,
  author = {Anu Pradhan and Alexandra Ortan and Apurv Verma and Madhavan Seshadri},
  title = {{LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generation}},
  year = {2025},
  eprint = {2509.12382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12382},
}

Loading related content...

Figure