TL;DR
When using Large Language Models (LLMs) to judge the quality of other models' answers in pairwise comparisons, their verdicts frequently depend on which answer is shown first or second—a phenomenon known as position bias. This study presents a systematic framework evaluating 12 leading LLM judges over 100,000 instances, revealing that position bias is not random noise, varies heavily across tasks and models, and is primarily driven by how close the two candidate solutions are in quality rather than prompt length.
What Problem Does This Paper Address?
The "LLM-as-a-Judge" paradigm has become the standard for scalable model evaluation, particularly via pairwise comparisons (asking a judge model whether Answer A or Answer B is superior). However, evaluator models frequently suffer from position bias: favoring a candidate simply based on its position in the prompt.
Prior research often treated this bias as an unpredictable anomaly, conflating it with run-to-run stochasticity, and largely ignored the direction of the bias—whether a model systematically leans toward the first option (primacy) or the last option (recency). Furthermore, it remained unclear what underlying factors (e.g., prompt length, model families, or solution quality) actually trigger position bias.
Methods
The authors introduce a unified evaluation framework built around three core metrics:
- Repetition Stability (RS): Measures how consistently an LLM judge makes the exact same choice when evaluated across multiple repeated trials with identical prompts (temperature = 1.0). This isolates systematic bias from random sampling noise.
- Position Consistency (PC): The standard metric calculating how often a judge picks the same winning solution after the order of Candidate 1 and Candidate 2 is swapped.
- Preference Fairness (PF): A novel, normalized score $\in [-1, 1]$ quantifying directional bias. A score of $0$ indicates balanced fairness (or perfect consistency), while $+1$ denotes complete recency preference and $-1$ denotes complete primacy preference.
To understand what factors drive position bias, the authors applied bidirectional stepwise regression using the Akaike Information Criterion (AIC) across judge-level factors (model families), candidate-level factors (answer quality gap), and task-level factors (prompt and component lengths). They quantified the Answer Quality Gap ($\delta_q$) using overall win rates, treating inconsistent judgment pairs as ties to address data sparsity.
Datasets
The authors evaluated pairwise judgments on two established benchmarks:
- MTBench: 8 tasks (coding, extraction, humanities, math, reasoning, roleplay, STEM, writing) across 30 candidate models. Comparisons were run against a fixed baseline candidate (
vicuna-13b-v1.3) under a Three-Option mode (A, B, or tie). - DevBench: 14 detailed software engineering tasks (spanning UML class diagrams, UML sequence diagrams, and architecture design) across 10 candidate models, compared against human-written solutions under a Two-Option mode (A or B).
Key Insights and Ideas
- Position bias is systematic, not random luck: Capable commercial judges showed high repetition stability ($RS > 0.85$, with models like GPT-4, GPT-4o, and Claude-3.5-Sonnet exceeding $0.95$). Position bias reflects deterministic model behavior rather than sampling variance.
- The Answer Quality Gap is the primary culprit: Plotting win rates against position consistency revealed a distinct parabolic curve. When one answer is substantially better than the other, judges rarely exhibit position bias. Position bias peaks when candidate solutions are very close in quality (near a 50% win rate), where deciding a clear winner is inherently subjective.
- Consistency does not imply fairness: A judge can be highly consistent yet heavily unfair. For instance, on coding tasks, both GPT-4 and GPT-4o achieved top-tier consistency, but GPT-4 was significantly recency-biased while GPT-4o was strongly primacy-biased.
- Length plays a negligible role: Stepwise regression confirmed that input length and prompt length have virtually no statistically significant impact on position bias. Output length showed only a tiny effect, suggesting that widely reported "verbosity biases" are likely artifacts of longer answers being higher quality.
- Small open-source models struggle as zero-shot judges: Llama-2 (7B, 13B) and Llama-3 (8B) exhibited failure/error rates between 17% and 100% and near-random repetition stability ($RS < 0.5$), making them unreliable for zero-shot complex pairwise judging.
Key Results
- Top Performers: GPT-4 and Claude-3.5-Sonnet emerged as the most consistent judges. On MTBench, GPT-4 achieved a PC of $0.82 \pm 0.15$ and a PF of $0.02$, while Claude-3.5-Sonnet achieved a PC of $0.82 \pm 0.14$ and a PF of $0.01$.
- Benchmark Disagreement: Over 75% of instances achieved judge consensus across more than half the tested models. Instances of maximal disagreement—where judges split heavily and position bias spiked—accounted for less than $0.1\%$ of queries (0.03% on MTBench, 0.07% on DevBench).
Limitations and Future Work
- Task Scope & Open Weights: The study focused exclusively on pairwise comparisons with 12 closed-source models due to the failure of smaller open models. Evaluating larger open models (e.g., Llama-3-70B+) or fine-tuned judge models remains future work.
- Prompt Variations: Experiments utilized fixed system prompts from existing benchmarks; future work should test whether prompt framing (e.g., explicit warnings against position bias) directly reduces bias.
- Alternative Judging Formats: The framework has yet to be applied to pointwise scoring or listwise ranking evaluations.
Summary updated