Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs

Lin Shi, Weicheng Ma, Soroush Vosoughi

Abstract

LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.

Introduction

As Large Language Models (LLMs) proliferate, evaluating their outputs has become a significant bottleneck. While human assessment remains the gold standard, it lacks the scalability required for rapid development cycles. Consequently, the industry has pivoted toward "LLM-as-a-Judge," where a powerful model (e.g., GPT-4) evaluates the outputs of other models. However, these judges are susceptible to position bias: the tendency to prefer a candidate based on its placement (primacy or recency) rather than its content. This study provides a systematic framework to quantify and understand the drivers of this bias across 15 LLM judges.

Methodology and Metrics

The researchers conducted a massive evaluation involving 15 LLM judges (including the GPT, Claude, Gemini, and Llama families) across 150,000 instances. To move beyond anecdotal evidence, they introduced three key metrics to quantify behavior:

  1. Repetition Stability (RS): Measures whether a judge provides the same verdict when presented with an identical prompt multiple times. This distinguishes systematic bias from random "hallucinations."
  2. Position Consistency (PC): Evaluates if a judge maintains its preference when the order of candidate solutions is swapped.
  3. Preference Fairness (PF): A novel metric that identifies the specific direction of bias (e.g., a preference for the first vs. second option) while accounting for varying dataset sizes.
Figure 1: The evaluation framework for position bias, illustrating the transition from pairwise to list-wise comparisons and the associated metrics of stability and consistency.

The study utilized two major benchmarks: MTBench (8 general tasks) and DevBench (14 software development tasks), assessing approximately 40 different generator models.

Key Drivers of Position Bias

A central finding of this research is that position bias is not random. Capable judges like Claude-3.5-Sonnet and GPT-4 demonstrated near-perfect Repetition Stability (>0.95), suggesting that bias is a deterministic artifact of the model's reasoning or training.

The most significant predictor of bias is the "Quality Gap" between candidate solutions. When two answers differ significantly in quality, judges remain consistent. However, when the quality gap is narrow, judges often "default" to their inherent position bias to break the tie. Interestingly, factors like prompt or answer length were found to have a negligible impact on consistency compared to the quality of the solutions themselves.

Experimental Results and Familial Patterns

The study revealed that models from the same "family" exhibit similar judging patterns and biases, likely due to shared training data or architectural similarities.

Figure 4: Radar charts showing model family performance across domains and scatter plots correlating position consistency with win rates and preference fairness.
  • Top Judges: Claude-3.5-Sonnet, GPT-4, and Llama-3.3-70B emerged as the most stable evaluators across both benchmarks.
  • Task Dependency: Bias direction is not fixed. For example, the GPT family showed recency preference on MTBench but flipped to primacy preference on DevBench, highlighting that bias is highly task-dependent.
  • List-wise Fragility: Position bias becomes significantly more pronounced and complex in list-wise scenarios (ranking three or more models). Less capable models, such as GPT-3.5-Turbo, showed a massive drop in consistency when moving beyond pairwise comparisons.

Conclusion and Future Directions

This systematic investigation confirms that position bias is a consistent, non-random behavior primarily driven by task difficulty and the quality gap between candidates. While the most advanced models are more stable, they are not immune to positional heuristics in challenging scenarios. Future research should focus on developing methods to detect "hard-to-judge" instances—where bias is most likely to occur—to enhance the reliability of automated benchmarking and model selection.

Summary updated

Abstract

LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications.
BibTeX
@misc{shi2024judging,
  author = {Lin Shi and Weicheng Ma and Soroush Vosoughi},
  title = {{Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs}},
  year = {2024},
  eprint = {2406.07791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07791},
}
Figure