Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement

Ramaravind Kommiya Mothilal, Terry Jingchen Zhang, Raiyan Ahmed, Zhijing Jin, Shion Guha, Syed Ishtiaque AhmedarXiv 2026 · arXiv

TL;DR

This paper introduces Second-Order Bias (SOB): social bias that manifests not in an LLM's direct text generation, but in its judgment about biased content. By framing bias evaluation as a logical reasoning task grounded in epistemology, the authors reveal that safety-tuned LLMs consistently make unwarranted demographic assumptions about who accepts or rejects biased text, effectively bypassing standard safety guardrails.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly deployed as evaluators ("LLM-as-a-judge"), ensuring their neutrality is paramount. While substantial research focuses on first-order bias—whether a model generates toxic or stereotypical content—subtler biases in how models evaluate external content remain largely unstudied.

If an LLM judge relies on biased assumptions to evaluate social bias, it risks obscuring, misattributing, or legitimizing the very harms it is meant to police. This paper addresses this gap by formalizing Second-Order Bias (SOB) and providing a rigorous, ground-truth-free framework to detect and measure it.

Methods

The authors ground their framework in entitlement epistemology, conceptualizing social bias as misplaced epistemic entitlement—foundational assumptions that an agent treats as self-evident truths but lack empirical justification.

To measure SOB, they design a two-step diagnostic reasoning task:

  1. Logical Conditions: Given a biased text (e.g., "Women are too emotional to lead"), the model must judge to whom the text is acceptable or non-acceptable based on strict logical rules of inquiry.
  2. Demographic Attribution: If the model identifies a person, it must describe them using a fixed set of demographic variables (e.g., age, race, gender).

Crucially, because the input text contains absolutely no demographic context, the only epistemically warranted and unbiased response is "Unknown." If a model infers specific demographic traits for the acceptor or rejector, it commits a logical fallacy driven by implicit bias.

Figure 1: Our evaluation task for second-order bias. We ask to whom a biased text would be acceptable or not under logical conditions grounded in entitlement epistemology. Since no demographic information is provided, the epistemically warranted and unbiased re- sponse is Unknown (left). Any demographic attribution reflects the model’s misplaced entitlement to infer the demographics of who would accept or reject the biased text, indicating bias in the model’s judgment (right).

The authors define the SOB Score as the expected number of unwarranted demographic attributes returned per input, calculated as the product of the attribution rate (how often it guesses) and conditional intensity (how many attributes it guesses).

Datasets

The evaluation utilizes 2,457 biased examples sampled across five prominent NLP bias and toxicity datasets:

  • DynaB (dynamic hate speech detection)
  • ToxiGen (implicit toxicity)
  • HateCheck (functional tests for hate speech)
  • iSHate (implicit and explicit hate speech)
  • LingHate (implicitly abusive remarks)

These datasets span diverse target groups and forms of bias, including stereotyping, negative sentiment, and targeted hate.

Key Insights and Ideas

  • Safety Guardrails are Blind to Judgment Bias: Because the prompt asks the model to perform an analytical, descriptive evaluation rather than generate toxic text, refusal rates were near zero. The models willingly generated biased demographic profiles under the guise of "reasoning."
  • Reasoning Traces Can Amplify Bias: Introducing intermediary "thinking tokens" (reasoning/thinking model variants) yielded mixed results. While it mitigated bias in some models, it significantly amplified SOB in others (e.g., GPT-5.1), allowing them to construct elaborate, stereotypical justifications for their demographic attributions.
  • Surfacing Implicit Social Maps: When models failed the task, they relied on latent social structures. In acceptability tasks, they frequently mapped bias to dominant groups (e.g., attributing anti-woman bias to men). In non-acceptability tasks, they often fell into "epistemic exclusion," assuming the targeted group itself would accept the bias directed against them (e.g., mapping anti-Muslim text as acceptable to Muslims).

Key Results

The authors evaluated several open and closed frontier models (including Instruct "I" and Reasoning/Thinking "T" variants):

  • Widespread Failure: Almost all models exhibited significant SOB, particularly on the non-acceptability task, frequently failing to output the correct "Unknown" response.
  • Highest Bias: GPT-5.1-T demonstrated the highest vulnerability, averaging 5.33 unwarranted demographic attributes for acceptability and 4.94 for non-acceptability. Gemma 3-27B also showed high baseline bias.
  • Best Performers: Only Qwen-3.5-35B-I and OLMo-3.1-32B-T achieved near-zero SOB scores on the acceptability task, demonstrating a strong capacity to correctly default to "Unknown."
  • Target Asymmetry: Models do not distribute bias equally. Biased texts targeting Mexican and Immigrant groups triggered significantly higher demographic attributions than texts targeting disabled or LGBTQ+ individuals, indicating that models more aggressively "socially locate" certain types of discrimination.

Limitations and Future Work

  • Dataset Constraints: The framework inherits the underlying limitations and operationalization ambiguities of the source datasets (e.g., ToxiGen's synthetic generations).
  • Lack of Intersectionality: The current setup evaluates only one primary target group per instance, ignoring multi-target or intersectional dynamics.
  • Fixed Demographic Schema: The attributes were restricted to a rigid, U.S. Census-style dictionary, which may not capture broader or context-specific global identities.
  • Future Directions: The authors suggest establishing human baselines for this task and expanding the framework to multilingual settings and more fluid demographic categories.

Summary updated

Abstract

Evaluations of social bias in LLMs largely focus on whether models generate or imply biased content. However, as LLMs are increasingly used as judges of bias, they may exhibit social biases in subtler ways in how they evaluate biased content, which current methods do not systematically capture. We call this second-order bias: social bias in an LLM's judgment about social bias, which we evaluate through a novel, philosophically grounded reasoning task. Drawing on entitlement epistemology, we conceptualize bias as misplaced foundational knowledge that shapes an agent's rational inquiry, and derive a logical reasoning task for LLMs to judge to whom a biased text is acceptable or non-acceptable. We develop two simple metrics to measure how biased LLM judges are in inferring demographics for acceptability without sufficient support, and how these inferences vary across groups targeted by biased texts. Evaluating open and closed models, we find that our task evades safety guardrails by surfacing bias in model judgment. It varies systematically across target groups, reflects implicit social maps, and shows how models are still triggered by demographic labels. Our work points to the need for LLM bias evaluation in judgment tasks and broadly, for more theoretically grounded approaches to bias evaluation in NLP. We release our code and model responses at https://github.com/uofthcdslab/second-order-bias.
BibTeX
@misc{mothilal2026evaluating,
  author = {Ramaravind Kommiya Mothilal and Terry Jingchen Zhang and Raiyan Ahmed and Zhijing Jin and Shion Guha and Syed Ishtiaque Ahmed},
  title = {{Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement}},
  year = {2026},
  eprint = {2606.17506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.17506},
}

Loading related content...

Figure