Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases

Jin Huang, Yuhang Yan, Liu Liu, Yixin Wan, Wenxuan Wang, Kai‐Wei Chang, Michael R. Lyu

Abstract

Recent failures such as Google Gemini generating people of color in Nazi-era uniforms illustrate how AI outputs can be factually plausible yet socially harmful. AI models are increasingly evaluated for "fairness," yet existing benchmarks often conflate two fundamentally different dimensions: factual correctness and normative fairness. A model may generate responses that are factually accurate but socially unfair, or conversely, appear fair while distorting factual reality. We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation. We introduce Fact-or-Fair, a benchmark with (i) objective queries aligned with descriptive, fact-based judgments, and (ii) subjective queries aligned with normative, fairness-based judgments. Our queries are constructed from 19 statistics and are grounded in cognitive psychology, drawing on representativeness bias, attribution bias, and ingroup-outgroup bias to explain why models often misalign fact and fairness. Experiments across ten frontier models reveal different levels of fact-fair trade-offs. By reframing fairness evaluation, we provide both a new theoretical lens and a practical benchmark to advance the responsible model assessments.

Introduction: The Tension Between Fact and Fairness

In early 2024, the AI community witnessed a significant controversy when Google’s Gemini model generated historically inaccurate images, such as Black Vikings, in an attempt to provide diverse outputs. This incident highlighted a fundamental tension in AI development: the conflation of fact and fairness. Existing benchmarks often fail to distinguish between these two dimensions. A model might be "fair" by being diverse but "unfactual" by ignoring historical reality. Conversely, a model might be "factual" about a specific statistic (e.g., crime rates) but "unfair" if it uses that statistic to unfairly judge an individual—a cognitive error known as representativeness bias.

We argue that for AI to be truly responsible, it must be able to provide accurate world knowledge while maintaining neutrality and avoiding stereotypes in subjective scenarios. Identifying the boundary between fact and fair is essential for meaningful fairness evaluation.

Methodology: The Fact-or-Fair Benchmark

The researchers developed FACT-OR-FAIR, a two-pronged evaluation framework designed to separate world knowledge from social equity. The benchmark is built on 19 key U.S. societal indicators spanning economic, social, and health dimensions, with data sourced from authoritative bodies like the Bureau of Labor Statistics and the CDC.

Figure 1: Comparison of objective queries (requiring accuracy) and subjective queries (requiring fairness) for LLMs and T2I models.

The framework utilizes two types of queries:

  1. Objective Queries: These test a model's factual accuracy ($S_{fact}$). Models are asked direct questions based on real-world statistics, such as "Which gender has a higher life expectancy in the U.S.?"
  2. Subjective Queries: These test a model's fairness ($S_{fair}$) by placing it in scenarios where it must make a judgment about individuals. Even if a model knows a group-level statistic, it should remain neutral when evaluating a specific person to avoid stereotyping.

Theoretical Grounding: Cognitive Biases

To make these tests robust, the subjective queries are grounded in three specific cognitive biases from psychology:

  • Representativeness Bias: The tendency to assume an individual perfectly conforms to their group's average traits.
  • Attribution Error: Overestimating internal traits while underestimating situational factors when explaining behavior.
  • In-group/Out-group Bias: The tendency to favor one's own group while being more critical of others.

Key Results and the "Factuality Tax"

The researchers evaluated ten frontier models, including GPT-4o, Gemini-1.5-Pro, and DALL-E 3. Several key insights emerged:

  • The Factuality Tax: There is a formal trade-off between accuracy and diversity. As a model becomes more "accurate" (concentrating its answers on the ground-truth group), its entropy naturally decreases.
  • T2I Models are Fact-Blind: Text-to-Image (T2I) models like Midjourney and DALL-E 3 perform significantly worse than Large Language Models (LLMs) on factual queries, often approaching the performance of random guessing.
  • The Gender vs. Race Gap: LLMs are remarkably good at gender-related facts (scoring ~90% accuracy) but struggle significantly with race-related statistics (scoring as low as ~40-50%). This suggests that "safety" tuning may be suppressing factual knowledge about race to avoid appearing biased.
  • Susceptibility to Context: Models are highly susceptible to representativeness bias. When provided with a group statistic in the prompt, fairness scores dropped across the board, with some models seeing a 10-20% increase in biased decision-making.

Conclusion

By reframing fairness evaluation through the lens of cognitive psychology, the Fact-or-Fair benchmark provides a new theoretical and practical framework for model assessment. While models like GPT-4o and DALL-E 3 lead the field, the persistent struggle with race-related facts and susceptibility to contextual bias highlights the need for more nuanced alignment strategies that respect both factual reality and social equity. Future work will expand this benchmark to include intersectional identities and a broader geographic scope beyond the United States.

Summary updated

Abstract

Recent failures such as Google Gemini generating people of color in Nazi-era uniforms illustrate how AI outputs can be factually plausible yet socially harmful.AI models are increasingly evaluated for "fairness," yet existing benchmarks often conflate two fundamentally different dimensions: factual correctness and normative fairness.A model may generate responses that are factually accurate but socially unfair, or conversely, appear fair while distorting factual reality.We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation.We introduce FACT-OR-FAIR, a benchmark with (i) objective queries aligned with descriptive, factbased judgments, and (ii) subjective queries aligned with normative, fairness-based judgments.Our queries are constructed from 19 statistics and are grounded in cognitive psychology, drawing on representativeness bias, attribution bias, and ingroup-outgroup bias to explain why models often misalign fact and fairness.Experiments across ten frontier models reveal different levels of fact-fair trade-offs.By reframing fairness evaluation, we provide both a new theoretical lens and a practical benchmark to advance the responsible model assessments.Our test suite is publicly available at https: //github.com/uclanlp/Fact-or-Fair.
BibTeX
@misc{huang2025where,
  author = {Jin Huang and Yuhang Yan and Liu Liu and Yixin Wan and Wenxuan Wang and Kai‐Wei Chang and Michael R. Lyu},
  title = {{Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases}},
  year = {2025},
  eprint = {2502.05849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.05849},
}
Figure