TL;DR
This paper introduces Second-Order Bias (SOB): social bias that manifests not in an LLM's direct text generation, but in its judgment about biased content. By framing bias evaluation as a logical reasoning task grounded in epistemology, the authors reveal that safety-tuned LLMs consistently make unwarranted demographic assumptions about who accepts or rejects biased text, effectively bypassing standard safety guardrails.
What Problem Does This Paper Address?
As Large Language Models (LLMs) are increasingly deployed as evaluators ("LLM-as-a-judge"), ensuring their neutrality is paramount. While substantial research focuses on first-order bias—whether a model generates toxic or stereotypical content—subtler biases in how models evaluate external content remain largely unstudied.
If an LLM judge relies on biased assumptions to evaluate social bias, it risks obscuring, misattributing, or legitimizing the very harms it is meant to police. This paper addresses this gap by formalizing Second-Order Bias (SOB) and providing a rigorous, ground-truth-free framework to detect and measure it.
Methods
The authors ground their framework in entitlement epistemology, conceptualizing social bias as misplaced epistemic entitlement—foundational assumptions that an agent treats as self-evident truths but lack empirical justification.
To measure SOB, they design a two-step diagnostic reasoning task:
- Logical Conditions: Given a biased text (e.g., "Women are too emotional to lead"), the model must judge to whom the text is acceptable or non-acceptable based on strict logical rules of inquiry.
- Demographic Attribution: If the model identifies a person, it must describe them using a fixed set of demographic variables (e.g., age, race, gender).
Crucially, because the input text contains absolutely no demographic context, the only epistemically warranted and unbiased response is "Unknown." If a model infers specific demographic traits for the acceptor or rejector, it commits a logical fallacy driven by implicit bias.
The authors define the SOB Score as the expected number of unwarranted demographic attributes returned per input, calculated as the product of the attribution rate (how often it guesses) and conditional intensity (how many attributes it guesses).
Datasets
The evaluation utilizes 2,457 biased examples sampled across five prominent NLP bias and toxicity datasets:
- DynaB (dynamic hate speech detection)
- ToxiGen (implicit toxicity)
- HateCheck (functional tests for hate speech)
- iSHate (implicit and explicit hate speech)
- LingHate (implicitly abusive remarks)
These datasets span diverse target groups and forms of bias, including stereotyping, negative sentiment, and targeted hate.
Key Insights and Ideas
- Safety Guardrails are Blind to Judgment Bias: Because the prompt asks the model to perform an analytical, descriptive evaluation rather than generate toxic text, refusal rates were near zero. The models willingly generated biased demographic profiles under the guise of "reasoning."
- Reasoning Traces Can Amplify Bias: Introducing intermediary "thinking tokens" (reasoning/thinking model variants) yielded mixed results. While it mitigated bias in some models, it significantly amplified SOB in others (e.g., GPT-5.1), allowing them to construct elaborate, stereotypical justifications for their demographic attributions.
- Surfacing Implicit Social Maps: When models failed the task, they relied on latent social structures. In acceptability tasks, they frequently mapped bias to dominant groups (e.g., attributing anti-woman bias to men). In non-acceptability tasks, they often fell into "epistemic exclusion," assuming the targeted group itself would accept the bias directed against them (e.g., mapping anti-Muslim text as acceptable to Muslims).
Key Results
The authors evaluated several open and closed frontier models (including Instruct "I" and Reasoning/Thinking "T" variants):
- Widespread Failure: Almost all models exhibited significant SOB, particularly on the non-acceptability task, frequently failing to output the correct "Unknown" response.
- Highest Bias: GPT-5.1-T demonstrated the highest vulnerability, averaging 5.33 unwarranted demographic attributes for acceptability and 4.94 for non-acceptability. Gemma 3-27B also showed high baseline bias.
- Best Performers: Only Qwen-3.5-35B-I and OLMo-3.1-32B-T achieved near-zero SOB scores on the acceptability task, demonstrating a strong capacity to correctly default to "Unknown."
- Target Asymmetry: Models do not distribute bias equally. Biased texts targeting Mexican and Immigrant groups triggered significantly higher demographic attributions than texts targeting disabled or LGBTQ+ individuals, indicating that models more aggressively "socially locate" certain types of discrimination.
Limitations and Future Work
- Dataset Constraints: The framework inherits the underlying limitations and operationalization ambiguities of the source datasets (e.g., ToxiGen's synthetic generations).
- Lack of Intersectionality: The current setup evaluates only one primary target group per instance, ignoring multi-target or intersectional dynamics.
- Fixed Demographic Schema: The attributes were restricted to a rigid, U.S. Census-style dictionary, which may not capture broader or context-specific global identities.
- Future Directions: The authors suggest establishing human baselines for this task and expanding the framework to multilingual settings and more fluid demographic categories.
Summary updated