Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs

Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, Soroush VosoughiSubmission to ICLR 2025 · Withdrawn

TL;DR

When using Large Language Models (LLMs) to judge the quality of other models' answers in pairwise comparisons, their verdicts frequently depend on which answer is shown first or second—a phenomenon known as position bias. This study presents a systematic framework evaluating 12 leading LLM judges over 100,000 instances, revealing that position bias is not random noise, varies heavily across tasks and models, and is primarily driven by how close the two candidate solutions are in quality rather than prompt length.

What Problem Does This Paper Address?

The "LLM-as-a-Judge" paradigm has become the standard for scalable model evaluation, particularly via pairwise comparisons (asking a judge model whether Answer A or Answer B is superior). However, evaluator models frequently suffer from position bias: favoring a candidate simply based on its position in the prompt.

Prior research often treated this bias as an unpredictable anomaly, conflating it with run-to-run stochasticity, and largely ignored the direction of the bias—whether a model systematically leans toward the first option (primacy) or the last option (recency). Furthermore, it remained unclear what underlying factors (e.g., prompt length, model families, or solution quality) actually trigger position bias.

Methods

The authors introduce a unified evaluation framework built around three core metrics:

  1. Repetition Stability (RS): Measures how consistently an LLM judge makes the exact same choice when evaluated across multiple repeated trials with identical prompts (temperature = 1.0). This isolates systematic bias from random sampling noise.
  2. Position Consistency (PC): The standard metric calculating how often a judge picks the same winning solution after the order of Candidate 1 and Candidate 2 is swapped.
  3. Preference Fairness (PF): A novel, normalized score $\in [-1, 1]$ quantifying directional bias. A score of $0$ indicates balanced fairness (or perfect consistency), while $+1$ denotes complete recency preference and $-1$ denotes complete primacy preference.

To understand what factors drive position bias, the authors applied bidirectional stepwise regression using the Akaike Information Criterion (AIC) across judge-level factors (model families), candidate-level factors (answer quality gap), and task-level factors (prompt and component lengths). They quantified the Answer Quality Gap ($\delta_q$) using overall win rates, treating inconsistent judgment pairs as ties to address data sparsity.

Datasets

The authors evaluated pairwise judgments on two established benchmarks:

  • MTBench: 8 tasks (coding, extraction, humanities, math, reasoning, roleplay, STEM, writing) across 30 candidate models. Comparisons were run against a fixed baseline candidate (vicuna-13b-v1.3) under a Three-Option mode (A, B, or tie).
  • DevBench: 14 detailed software engineering tasks (spanning UML class diagrams, UML sequence diagrams, and architecture design) across 10 candidate models, compared against human-written solutions under a Two-Option mode (A or B).

Key Insights and Ideas

  • Position bias is systematic, not random luck: Capable commercial judges showed high repetition stability ($RS > 0.85$, with models like GPT-4, GPT-4o, and Claude-3.5-Sonnet exceeding $0.95$). Position bias reflects deterministic model behavior rather than sampling variance.
  • The Answer Quality Gap is the primary culprit: Plotting win rates against position consistency revealed a distinct parabolic curve. When one answer is substantially better than the other, judges rarely exhibit position bias. Position bias peaks when candidate solutions are very close in quality (near a 50% win rate), where deciding a clear winner is inherently subjective.
  • Consistency does not imply fairness: A judge can be highly consistent yet heavily unfair. For instance, on coding tasks, both GPT-4 and GPT-4o achieved top-tier consistency, but GPT-4 was significantly recency-biased while GPT-4o was strongly primacy-biased.
  • Length plays a negligible role: Stepwise regression confirmed that input length and prompt length have virtually no statistically significant impact on position bias. Output length showed only a tiny effect, suggesting that widely reported "verbosity biases" are likely artifacts of longer answers being higher quality.
  • Small open-source models struggle as zero-shot judges: Llama-2 (7B, 13B) and Llama-3 (8B) exhibited failure/error rates between 17% and 100% and near-random repetition stability ($RS < 0.5$), making them unreliable for zero-shot complex pairwise judging.

Key Results

  • Top Performers: GPT-4 and Claude-3.5-Sonnet emerged as the most consistent judges. On MTBench, GPT-4 achieved a PC of $0.82 \pm 0.15$ and a PF of $0.02$, while Claude-3.5-Sonnet achieved a PC of $0.82 \pm 0.14$ and a PF of $0.01$.
  • Benchmark Disagreement: Over 75% of instances achieved judge consensus across more than half the tested models. Instances of maximal disagreement—where judges split heavily and position bias spiked—accounted for less than $0.1\%$ of queries (0.03% on MTBench, 0.07% on DevBench).

Limitations and Future Work

  • Task Scope & Open Weights: The study focused exclusively on pairwise comparisons with 12 closed-source models due to the failure of smaller open models. Evaluating larger open models (e.g., Llama-3-70B+) or fine-tuned judge models remains future work.
  • Prompt Variations: Experiments utilized fixed system prompts from existing benchmarks; future work should test whether prompt framing (e.g., explicit warnings against position bias) directly reduces bias.
  • Alternative Judging Formats: The framework has yet to be applied to pointwise scoring or listwise ranking evaluations.

Summary updated

Abstract

LLM-as-a-Judge presents a promising alternative to human evaluators across various tasks, but inherent biases, especially position bias — a tendency to favor solutions based on their position in the prompt — have compromised its effectiveness. Our study introduces a systematic framework to examine position bias in pairwise comparisons, focusing on repetition stability, position consistency, and preference fairness. This research significantly contributes to the field by introducing new concepts for understanding position bias and providing a multi-dimensional framework for evaluations. We conducted experiments with 12 LLM judges across MTBench and DevBench, covering 22 tasks and approximately 40 solution-generating models — candidates, resulting in over 100,000 evaluation instances. Our findings confirm that position bias in capable LLM judges is not due to random chances, along with notable variations observed across judges and tasks. Moreover, position bias is weakly influenced by the length of prompt components but significantly impacted by the quality gap between solutions. These insights can help optimize judge model selections, improve benchmark design, and inform future research on debiasing strategies, ultimately enhancing the reliability of LLM judges.
BibTeX
@misc{shi2024judging,
  author = {Lin Shi and Chiyu Ma and Wenhua Liang and Weicheng Ma and Soroush Vosoughi},
  title = {{Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs}},
  year = {2024},
  eprint = {2406.07791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07791},
}

Loading related content...

Figure