Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge

Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, Soroush Vosoughi

Abstract

Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, Soroush Vosoughi. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

2025.

Introduction

The "LLM-as-a-Judge" paradigm—using powerful models like GPT-4 to evaluate the outputs of other models—has become the industry standard for automated evaluation. However, these judges are susceptible to position bias, where the order of presented solutions significantly influences the final verdict. This study provides a systematic analysis of 15 major LLMs across 22 tasks, revealing that position bias is a consistent, non-random behavior primarily driven by the "quality gap" between candidate answers rather than superficial factors like prompt length.

Methodology

The researchers conducted an extensive study involving over 150,000 evaluation instances. They utilized 15 LLM judges (including GPT-4, Claude 3, Gemini 1.5, and Llama 3) to evaluate solutions from approximately 40 candidate models on benchmarks like MTBench and DevBench.

To quantify bias, the authors introduced a refined framework featuring three metrics: Repetition Stability (RS) to distinguish systematic bias from random noise, Position Consistency (PC) to measure verdict stability across swapped positions, and Preference Fairness (PF) to quantify the direction of bias (primacy vs. recency).

Framework for measuring position bias and preference fairness in LLM-as-a-judge evaluations.

Key Insights: The Quality Gap Driver

A central finding of the study is that position bias is systematic. High-performing judges like Claude 3.5 Sonnet and GPT-4o exhibit near-perfect Repetition Stability (>0.95), indicating that their biased decisions are deliberate rather than stochastic.

The primary predictor of bias is the quality gap between candidates. When one answer is clearly superior, judges are highly consistent. However, as the quality gap narrows, judges increasingly default to their inherent position bias. This is visualized in the relationship between consistency and win rate: consistency is highest when one model dominates and lowest when the competition is close (forming a U-shaped curve).

The U-shaped relationship between Position Consistency and Overall Win Rate, demonstrating that bias is most prevalent when candidate quality is similar.

Contrary to common assumptions, factors like prompt length or answer length showed weak correlations with bias. Instead, "familial bias" was observed, where models from the same family (e.g., GPT series) showed higher mutual agreement and similar judging patterns.

Performance and Scalability

  • Top Performers: Claude-3.5-Sonnet and GPT-4 emerged as the most robust judges, maintaining Position Consistency scores around 0.82.
  • List-wise Ranking: Moving from pairwise (A vs. B) to list-wise (ranking 3+ models) comparisons significantly degrades consistency, especially for weaker models like GPT-3.5-Turbo.
  • Task-Dependent Bias: The direction of bias can vary by task. For instance, GPT-4 favored the second option (recency) on MTBench but the first option (primacy) on DevBench.

Limitations and Future Directions

The study primarily focused on closed-source models, limiting the ability to analyze internal mechanisms like attention weights. Future work should focus on developing "hard" benchmarks with narrow quality gaps to stress-test judges and exploring "meta-judging" strategies that aggregate multiple model opinions to mitigate individual biases.

Summary updated

BibTeX
@misc{shi2024judging,
  author = {Lin Shi and Chiyu Ma and Wenhua Liang and Xingjian Diao and Weicheng Ma and Soroush Vosoughi},
  title = {{Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge}},
  year = {2024},
  eprint = {2406.07791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07791},
}
Figure