Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

Rajarshi Haldar, Julia HockenmaierEMNLP 2025

TL;DR

Large Language Models (LLMs) are increasingly used to evaluate Natural Language Generation (NLG) outputs, but this paper reveals a significant problem: LLM judges exhibit low self-reliability. This means they often give inconsistent ratings for the same input across different runs, even with identical prompts, undermining their trustworthiness as evaluators.

What Problem Does This Paper Address?

As NLG systems become more sophisticated and widely deployed, reliable evaluation methods are crucial. Traditional metrics like BLEU struggle with open-ended tasks like summarization or dialogue. LLM-as-a-judge frameworks have emerged as a promising, scalable alternative, where LLMs are prompted to score or rank generated text. However, most studies focus on agreement between LLM judges and human judges (inter-rater reliability) or between different LLM judges, neglecting a critical aspect: self-reliability (or intra-rater reliability). This refers to how consistent a single judge is with itself over multiple evaluations of the same item. Without this, we can't be sure if an LLM's rating reflects genuine quality assessment or just random variation.

Methods

The authors investigate LLM self-reliability across three NLG tasks:

  1. SummaC: A binary classification task for factual consistency in summarization.
  2. SummEval: A summarization benchmark where LLMs rate summaries on a 1-5 Likert scale for coherence, consistency, fluency, and relevance.
  3. MT-Bench: A multi-turn conversational benchmark where LLMs judge which of two conversational agents is better.

For each benchmark, they prompt three LLMs (Llama 3.1, Deepseek-R1, and Qwen 3) to evaluate the same set of generations independently for three runs. They use Krippendorff's Alpha, a chance-corrected reliability metric, to measure self-reliability. They also explore the impact of disabling sampling (setting temperature to zero) and using few-shot or chain-of-thought prompting on reliability and performance.

Datasets

  • SummaC: A benchmark unifying six datasets (CoGenSumm, XSum-Faith, Polytope, FactCC, SummEval, and FRANK) for factual consistency detection in summarization.
  • SummEval: A summarization benchmark with 1700 examples, featuring human ratings on a 1-5 Likert scale for coherence, consistency, fluency, and relevance.
  • MT-Bench: A benchmark for evaluating multi-turn conversational and instruction-following abilities of LLMs, involving user prompts and LLM-generated responses.

Key Insights and Ideas

  • LLMs are inconsistent judges: The most striking finding is the consistently low self-reliability of LLM judges across various tasks and metrics. This suggests that LLM ratings, even for the same input, can vary significantly from one run to another.
  • Larger models are more reliable, but still fall short: While newer and larger models like Qwen 3 show better self-reliability than smaller ones like Llama 3.1, they still often fail to meet the commonly accepted threshold of 0.8 for good agreement.
  • Disabling sampling doesn't fully solve the problem: Turning off sampling to ensure identical outputs per run can actually hurt performance (accuracy against human judgments), indicating a trade-off between self-reliability and evaluation quality.
  • Subjectivity is a shared problem: The low agreement among human raters on certain metrics (especially relevance in SummEval) suggests that some aspects of NLG quality are inherently subjective, and this subjectivity likely contributes to LLM inconsistency as well.
  • Accuracy inflates agreement: Metrics like accuracy, commonly used in LLM-as-a-judge studies, can overestimate agreement because they don't account for chance agreement, unlike Krippendorff's Alpha.

Key Results

  • SummaC: LLM judges achieved low self-reliability, with Krippendorff's Alpha values often below 0.8. Qwen 3 showed the highest self-reliability (around 0.788), while Llama 3.1 was significantly lower (around 0.326).
  • SummEval: Self-reliability varied by metric. LLMs showed higher consistency on coherence and consistency but struggled with fluency.
  • MT-Bench: This task exhibited the lowest self-reliability among the three, with Qwen 3 achieving an Alpha of only 0.563.
  • Balanced Accuracy vs. Human Judgments: While LLMs showed some agreement with human judgments, the low self-reliability of the LLMs themselves acts as a ceiling on how well they can match human performance. Taking a majority vote across multiple LLM runs improved agreement with human judgments compared to a single run.

Limitations and Future Work

  • Human judge self-reliability: The paper acknowledges that they did not extensively study the self-reliability of human judges, which is crucial since human judgments are the gold standard. Future work should collect this data.
  • Task definition and subjectivity: It's unclear if the low LLM self-reliability is due to LLM limitations or if tasks/metrics themselves are poorly defined or inherently subjective, leading to fluctuating scores for both humans and LLMs.
  • Prompting strategies: While few-shot and CoT prompting were explored, they did not significantly improve self-reliability. Future work could investigate other prompting techniques or fine-tuning.
  • Model scope: The study used open-weight models. Comparing with the latest proprietary models (e.g., GPT-5, Claude 4) would provide a broader understanding of the phenomenon.
  • Reasoning traces: The paper did not explore the relationship between LLM reasoning traces and their self-reliability.

Summary updated

BibTeX
@inproceedings{haldar2025rating,
  author = {Rajarshi Haldar and Julia Hockenmaier},
  title = {{Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure