Fact-or-Fair: A Checklist for Behavioral Testing of AI Models on Fairness-Related Queries

Jen-tse Huang, Yuhang Yan, Linqi Liu, Yixin Wan, Wenxuan Wang, Kai-Wei Chang, Michael R. Lyu

Abstract

The generation of incorrect images, such as depictions of people of color in Nazi-era uniforms by Gemini, frustrated users and harmed Google's reputation, motivating us to investigate the relationship between accurately reflecting factuality and promoting diversity and equity. In this study, we focus on 19 real-world statistics collected from authoritative sources. Using these statistics, we develop a checklist comprising objective and subjective queries to analyze behavior of large language models (LLMs) and text-to-image (T2I) models. Objective queries assess the models' ability to provide accurate world knowledge. In contrast, the design of subjective queries follows a key principle: statistical or experiential priors should not be overgeneralized to individuals, ensuring that models uphold diversity. These subjective queries are based on three common human cognitive errors that often result in social biases. We propose metrics to assess factuality and fairness, and formally prove the inherent trade-off between these two aspects. Results show that GPT-4o and DALL-E 3 perform notably well among six LLMs and four T2I models. Our code is publicly available at https://github.com/uclanlp/Fact-or-Fair.

Introduction

In early 2024, the AI community witnessed a significant controversy when Google’s Gemini model generated historically inaccurate images, such as "Black Vikings," in an attempt to promote diversity. This incident underscored a fundamental tension in AI development: the conflict between Factual Correctness and Normative Fairness. While diversity is a crucial social value, its pursuit should not come at the cost of distorting objective reality. Conversely, a model that strictly adheres to historical statistics in subjective scenarios may inadvertently perpetuate harmful stereotypes.

The FACT-OR-FAIR benchmark addresses this gap by distinguishing between objective world knowledge and subjective social scenarios. The authors argue that "bias" cannot be evaluated without first defining the boundary where facts end and fairness (as a social value) begins.

The FACT-OR-FAIR Framework

The researchers introduced a framework that evaluates models using two distinct query types based on 19 real-world US statistics (2019–2024) across economic, social, and health dimensions.

  1. Objective Queries: These test the model's ability to provide accurate world knowledge (e.g., "Which group has the highest crime rate in the US?"). Performance is measured by Accuracy ($S_{fact}$).
  2. Subjective Queries: These present scenarios with no single "correct" demographic answer (e.g., "Which of these four patients is most likely to have the flu?"). Performance is measured by Fairness ($S_{fair}$), using entropy for diversity and KL divergence for consistency.
Examples of subjective queries for LLMs and T2I models, illustrating the distinction between fair and unfair model responses.

To further probe model behavior, the framework includes Cognitive Bias Contexts designed to trigger human-like psychological errors: Representativeness Bias (overgeneralizing group statistics to individuals), Attribution Error (blaming internal traits rather than circumstances), and In-group/Out-group Bias (observing if models favor identities they are instructed to assume).

Experimental Setup

The study evaluated six LLMs (including GPT-4o, Gemini-1.5-Pro, Qwen-2.5, and LLaMA-3.2) and four T2I models (including DALL-E 3 and Midjourney). The dataset utilized authoritative sources such as the BLS, Census Bureau, FBI, and CDC to ensure the ground truth for objective queries was robust.

Key Findings

The Fact-Fair Trade-off

The study formally proves and empirically demonstrates a "tax" on diversity. Models that excel at objective factuality often struggle to remain neutral in subjective scenarios. Conversely, models heavily aligned toward diversity often fail basic factual checks.

LLMs and Human Psychology

LLMs exhibit cognitive biases similar to humans. They are particularly susceptible to Representativeness Bias; when provided with a group statistic, they often fail to treat an individual from that group as a "blank slate" in subjective scenarios. Baseline fairness scores ($S_{fair}$ > 97) dropped significantly when these bias contexts were introduced (e.g., GPT-3.5 dropped to 91.46).

T2I Models and the "Woke" Filter

Text-to-image models were found to be largely "fact-blind," with accuracy scores on objective queries ($S_{fact}$ 36–44%) often nearing random chance. Furthermore, a "woke filter" effect was observed in some LLMs like GPT-4o, which showed high accuracy on gender-related facts (~90%) but significantly lower accuracy on race-related facts (~55%), suggesting that safety alignment may be suppressing factual data regarding race to avoid appearing biased.

Conclusion

GPT-4o and DALL-E 3 emerged as the top performers, balancing factuality and fairness more effectively than their peers. However, the study highlights significant gaps, particularly in T2I models' world knowledge and the susceptibility of all models to cognitive biases. Future work will aim to expand the geographic scope beyond the US and incorporate more granular demographic categories to better capture intersectional identities.

Summary updated

Abstract

The generation of incorrect images, such as depictions of people of color in Nazi-era uniforms by Gemini, frustrated users and harmed Google's reputation, motivating us to investigate the relationship between accurately reflecting factuality and promoting diversity and equity. In this study, we focus on 19 real-world statistics collected from authoritative sources. Using these statistics, we develop a checklist comprising objective and subjective queries to analyze behavior of large language models (LLMs) and text-to-image (T2I) models. Objective queries assess the models' ability to provide accurate world knowledge. In contrast, the design of subjective queries follows a key principle: statistical or experiential priors should not be overgeneralized to individuals, ensuring that models uphold diversity. These subjective queries are based on three common human cognitive errors that often result in social biases. We propose metrics to assess factuality and fairness, and formally prove the inherent trade-off between these two aspects. Results show that GPT-4o and DALL-E 3 perform notably well among six LLMs and four T2I models. Our code is publicly available at https://github.com/uclanlp/Fact-or-Fair.
BibTeX
@misc{huang2025fact,
  author = {Jen-tse Huang and Yuhang Yan and Linqi Liu and Yixin Wan and Wenxuan Wang and Kai-Wei Chang and Michael R. Lyu},
  title = {{Fact-or-Fair: A Checklist for Behavioral Testing of AI Models on Fairness-Related Queries}},
  year = {2025},
  eprint = {2502.05849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.05849},
}
Figure