Self-Preference Bias in LLM-as-a-Judge

Koki Wataoka, Tsubasa Takahashi, Ryokan RiSubmission to ICLR 2025 · Rejected

TL;DR

Using LLMs as judges often leads to self-preference bias, where models favor their own generated responses over others. This paper introduces an algorithmic fairness-based metric to quantify this bias and demonstrates that it is largely driven by text perplexity—models systematically prefer familiar, low-perplexity outputs rather than intentionally favoring their own identity.

What Problem Does This Paper Address?

The "LLM-as-a-judge" paradigm is widely used to benchmark conversational models, but evaluators can introduce severe biases. A notable concern is self-preference bias: an LLM overrating its own outputs compared to human ground truth. Previous work lacked standard quantitative metrics for pairwise evaluations and failed to isolate the underlying cause of why models prefer their own text.

Methods

  • Fairness-Based Metric: The authors adapt the Equal Opportunity fairness criterion to pairwise evaluation. The metric calculates the difference in true positive rates (evaluator agreement with humans) when the candidate text is generated by the evaluator versus another model: $$\text{Bias} = P(Y'=1 \mid S=1, Y=1) - P(Y'=1 \mid S=0, Y=1)$$
  • Perplexity Analysis: To probe the mechanism, the authors evaluate whether preference correlates with conditioned log-perplexity, comparing how winning rates shift as perplexity varies between candidate answers.
  • Position Bias Mitigation: Swapping response order (A/B) to average out positional preferences.

Datasets

  • Chatbot Arena Dataset: Consists of 33,000 real-world user queries paired with two LLM-generated responses and human preference annotations (model_a, model_b, or tie). No new benchmark datasets were introduced.

Key Insights and Ideas

  • Perplexity as the Root Driver: Models assign significantly higher win rates to lower-perplexity responses relative to humans, regardless of which model actually generated the response.
  • Self-Preference is an Artifact of Familiarity: LLMs naturally have lower perplexity on text sampled from their own distribution. Thus, self-preference is predominantly a symptom of language familiarity rather than explicit self-recognition.
  • Advanced Models Show Stronger Self-Preference: Frontier models like GPT-4 adhere strictly to distinct stylistic patterns and policies, penalizing alternate valid approaches that diverge from their training distribution.

Key Results

  • Bias Across Models: GPT-4 exhibited the highest self-preference score (0.520 under Equal Opportunity, 0.749 under Demographic Parity), followed by Vicuna-13B (0.250) and Koala-13B (0.150). Smaller models (e.g., Dolly-v2-12B at -0.02, StableLM-7B at -0.05) showed slight negative bias, undervaluing their own outputs.
  • Perplexity Sensitivity: For open-weight evaluators (Vicuna, Koala, and Llama families), preference curves tracked perplexity differences far more steeply than human evaluators did.
  • Conditioned Disparity: When conditioned on perplexity differences, an LLM's preference rate for its own outputs mirrored its preference rate for other models' outputs.

Limitations and Future Work

  • Closed API Restrictions: Token-level log probabilities and perplexities could not be directly computed for proprietary models like GPT-4 and GPT-3.5-Turbo.
  • Mitigation Strategies: While the authors suggest multi-model ensembling and weighting judgments inversely by perplexity, these solutions were proposed conceptually without full empirical testing.

Summary updated

Abstract

Automated evaluation leveraging large language models (LLMs), commonly referred to as LLM evaluators or LLM-as-a-judge, has been widely used in measuring the performance of dialogue systems. However, the self-preference bias in LLMs has posed significant risks, including promoting specific styles or policies intrinsic to the LLMs. Despite the importance of this issue, there is a lack of established methods to measure the self-preference bias quantitatively, and its underlying causes are poorly understood. In this paper, we introduce a novel quantitative metric to measure the self-preference bias. Our experimental results demonstrate that GPT-4 exhibits a significant degree of self-preference bias. To explore the causes, we hypothesize that LLMs may favor outputs that are more familiar to them, as indicated by lower perplexity. We analyze the relationship between LLM evaluations and the perplexities of outputs. Our findings reveal that LLMs assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated. This suggests that the essence of the bias lies in perplexity and that the self-preference bias occurs because the LLMs' own outputs have lower perplexity.
BibTeX
@inproceedings{wataoka2024self,
  author = {Koki Wataoka and Tsubasa Takahashi and Ryokan Ri},
  title = {{Self-Preference Bias in LLM-as-a-Judge}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=tLZZZIgPJX},
}
Figure