Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, Xiangliang Zhang

Abstract

As Large Language Models (LLMs) increasingly serve as evaluators for other AI systems, they introduce hidden "prejudices" into the assessment process. This paper introduces CALM (Comprehensive Assessment of Language Model Judge Biases), a framework designed to quantify 12 distinct types of bias across major models. Our evaluation of leading judges, including GPT-4o and Claude 3.5, reveals significant vulnerabilities to non-substantive factors such as response length, positional ordering, and fabricated authority. We demonstrate that even high-performing models rely on cognitive heuristics that can compromise the integrity of AI evaluation.

Introduction

The industry is rapidly adopting the "LLM-as-a-Judge" paradigm, utilizing models like GPT-4 to evaluate AI-generated responses. While this approach offers significant advantages in speed and cost over human evaluation, it introduces a "black box" of judgment. Existing research has often focused on narrow bias types, such as position bias, leaving a gap in our understanding of the broader landscape of evaluative prejudices.

Without a systematic method to quantify these biases, there is a risk of creating a feedback loop where models are optimized for superficial traits—such as verbosity or sycophancy—rather than factual accuracy and logical consistency. This paper seeks to move beyond subjective comparisons by establishing an automated, principle-guided framework to stress-test AI judges.

Methodology: The CALM Framework

The authors propose CALM, which employs an "attack-and-detect" methodology to probe the robustness of LLM judges. The framework consists of three primary components:

  1. Systematic Categorization: The study identifies 12 key biases, including Verbosity (favoring length), Authority (favoring citations), Self-Enhancement (favoring self-generated content), and Position Bias.
  2. Automated Perturbation: CALM uses "principle-guided modification" (powered by GPT-4o) to inject specific biases into "raw" answers. This process ensures that the core factual correctness remains unchanged while stylistic or structural elements are manipulated to trigger specific biases.
  3. Quantification Metrics: Two primary metrics are used: Robustness Rate (RR), measuring how often a judge maintains its original correct judgment post-perturbation, and Consistency Rate (CR), assessing the stability of judgments under identical conditions.
Overview of the CALM framework, including the bias injection pipeline, datasets, and evaluation metrics.

The study utilizes a diverse set of data, including fact-related tasks (GSM8K, MATH, ScienceQA), alignment datasets (OpenOrca, Orca-DPO), and refinement-aware sets (TruthfulQA) to cover a wide spectrum of judging scenarios.

Experimental Results and Insights

The researchers evaluated six prominent models: ChatGPT, GPT-4-Turbo, GPT-4o, GLM-4, Claude-3.5-Sonnet, and Qwen2-72B.

Model Robustness Comparison

Claude-3.5-Sonnet emerged as the most resilient judge, particularly in resisting Authority and Diversity biases. Interestingly, the study found that model size and general capability do not always correlate with judging fairness. For instance, GPT-4-Turbo exhibited surprising inconsistency in judging emotional responses compared to its predecessors.

Radar chart illustrating the robustness of six major LLMs across 12 bias categories, highlighting strengths and vulnerabilities.

Key Bias Findings

  • Universal Verbosity Bias: Nearly all models showed a propensity to favor longer responses, even when the added length did not contribute to quality.
  • The Authority Trap: Models are easily "hacked" by the presence of citations. The Authority Bias results indicate that judges assign higher credibility to statements accompanied by references, even when those references are entirely fabricated.
  • Position and Sentiment: Position bias remains a significant hurdle, with models favoring the first-presented answer. Furthermore, Sentiment Bias often causes judges to flip their preference from a superior but "frustrated" answer to a lower-quality but "polite" one.
  • Fact vs. Alignment: Biases are significantly more pronounced in open-ended Alignment tasks. In factual tasks, the quality gap is often large enough to override bias, but in subjective scenarios, bias frequently dictates the outcome.

Discussion and Future Work

The findings underscore the necessity of "de-biasing" LLM judges. While CALM provides a comprehensive baseline, the evolving nature of LLM architectures suggests that new biases may emerge. The authors acknowledge limitations, such as the focus on English-language datasets, and suggest that cross-lingual bias remains an important area for future research.

Future efforts should focus on developing "protective prompts" and specialized fine-tuning techniques to ensure that AI judges prioritize logic, truth, and relevance over stylistic cues and positional advantages.

Summary updated

BibTeX
@inproceedings{ye2025justice,
  author = {Jiayi Ye and Yanbo Wang and Yue Huang and Dongping Chen and Qihui Zhang and Nuno Moniz and Tian Gao and Werner Geyer and Chao Huang and Pin-Yu Chen and Nitesh V. Chawla and Xiangliang Zhang},
  title = {{Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=3GTtZFiajM},
}
Figure