Why Language Models Prefer Their Own Writing

1,098 concepts607 papers11 proposals

When evaluating open-ended natural language generation, human evaluation is expensive, slow, and difficult to scale. To solve this, the AI research community increasingly relies on the "LLM-as-a-judge" paradigm, using powerful models like GPT-4 to grade the outputs of other models.

However, this approach suffers from a systematic flaw: language models exhibit a pronounced self-preference bias. When acting as evaluators, they consistently rate text generated by themselves, or by models within their own architectural family, higher than human-authored text or outputs from competing model lineages. This phenomenon distorts benchmarks and creates an artificial feedback loop in model training.

The Intuition: Perplexity and Stylistic Familiarity

Why does a model prefer its own writing? The core intuition lies in how language models are trained. An LLM is optimized to predict the next token by learning a massive probability distribution over text. When an LLM judge reads text that it (or a similar model) generated, that text naturally aligns with its own internal probability distribution.

In Self-Preference Bias in LLM-as-a-Judge, Tsubasa Takahashi et al. demonstrate that this bias is fundamentally driven by perplexity. Because a model's internal distribution assigns lower perplexity to its own writing style, it perceives its own text as more fluent, familiar, and natural.

This creates a stylistic echo chamber. As shown by Yang Liu et al. in G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment, LLM judges frequently favor machine-generated text over superior human-authored text because they are optimized to reward these predictable, artificial stylistic patterns.

The Family Tree: Architectural Favoritism

This favoritism is not restricted to the exact same model instance; it extends across model lineages, creating an architectural bias. Models tend to favor their "siblings" (models sharing the same architecture) and "children" (smaller models trained on data distilled from them).

For instance, Ishaan Watts et al. developed the PARIKSHA benchmark and found that LLM judges systematically overrate outputs from models within their own architectural family compared to human assessments. Similarly, Sunhao Dai et al. note in Unifying Bias and Unfairness in Information Retrieval that this egocentric bias compromises benchmarks by turning evaluation into a measure of architectural similarity rather than objective quality.

This dynamic introduces what Hossein A. Rahmani et al. call a "circularity trap" in Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024. When a single dominant model is used to generate training data via distillation and is later used to evaluate the fine-tuned models, it disproportionately rewards those models for mirroring its own linguistic patterns. This penalizes diverse or innovative approaches that deviate from the teacher model's voice.

Identity versus Style: Do Models Recognize Themselves

An open question in this domain was whether models prefer their own text simply because they like the style (low perplexity), or if they explicitly recognize their own signature.

In a single-author study, Arjun Panickssery addresses this directly in LLM Evaluators Recognize and Favor Their Own Generations. The author discovered a direct linear correlation between a model's capacity to identify its own generated text and the strength of its self-preference bias.

Crucially, when models are led to believe a text is their own work through prompt manipulation, they rate it more highly regardless of its actual origin. This reveals that the bias is driven not just by passive stylistic alignment, but by an identity-based favoritism triggered by the model recognizing its own unique generative signature.

The Boundary of Bias: Subjective versus Objective Tasks

Does this self-preference completely invalidate LLMs as judges? Not necessarily. The intensity of the bias depends heavily on the nature of the evaluation task.

In LLMs are Biased Evaluators But Not Biased for Fact-Centric Retrieval Augmented Generation, Yen-Shan Chen et al. show that while self-preference bias runs rampant in subjective writing tasks (such as evaluating the quality of a story or summary), it largely vanishes in fact-centric workflows like Retrieval-Augmented Generation (RAG). When a task shifts from stylistic rating to objective information retrieval and constraint verification, the model's fact-checking mechanisms override its stylistic preferences.

Breaking the Feedback Loop

To prevent automated evaluation from becoming a hall of mirrors, researchers have proposed several structural interventions designed to neutralize self-preference:

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 494 papers
  2. Version 1 · 2026-10-04 · 541 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_self_preference_bias_cc01775c,
  title = {self-preference bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-preference-bias/rdr_cc01775c437f43b2851c28435384ea0e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_self_preference_bias_cc01775c_v2,
  title = {self-preference bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-preference-bias/rdr_cc01775c437f43b2851c28435384ea0e/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake