Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources

Lasse Hyldig Hansen, Nikolaj Andersen, Jack Gallifant, Liam G. McCoy, James K. Stone, Nura Izath, Marcela Aguirre-Jerez, Danielle S. Bitterman, Judy Gichoya, Leo Anthony CeliarXiv 2024 · DOI · arXiv

Introduction

As Large Language Models (LLMs) like GPT-4 are increasingly integrated into healthcare for tasks such as clinical summarization and diagnosis, researchers have noted a disturbing trend: these models often produce outputs reflecting deep-seated racial and gender stereotypes. This paper seeks to quantify the "representational harms" within the massive, uncurated datasets used to train these models. By analyzing 1.2 trillion tokens of web-based data, the authors aim to determine if the biases seen in AI outputs are reflections of the internet's "collective consciousness" or if the models are amplifying existing disparities.

Methodology

The authors employed a large-scale textual co-occurrence analysis to measure how often specific diseases are mentioned alongside demographic markers. The process involved three key steps:

  1. Keyword Definition: Curating a dictionary for 18 specific diseases and various racial/gender groups using medical (MeSH) headings.
  2. Proximity Mapping: Using "token windows" (20, 100, 200, and 500 words) to distinguish between passing mentions and direct contextual associations.
  3. Comparative Baseline: Comparing text proportions against actual US population disease prevalence and documented GPT-4 biases.
Methodological workflow for counting keyword co-occurrences in large-scale text corpora.

The primary dataset was RedPajama-Data-1T, an open-source collection designed to replicate the training data for Meta’s LLaMA-1, encompassing web scrapes (C4, Common Crawl), scientific papers (arXiv), books, and Wikipedia.

Key Findings

The "White Default" and Racial Skew

A striking finding was the "white defaultism" in English text—the tendency to assume a subject is White unless specified otherwise. This leads to a skewed data profile where "Black" is used as a specific marker, while "White" remains invisible. Consequently, Black individuals are significantly overrepresented in discussions of nearly all analyzed diseases compared to real-world population statistics.

Quantitative Disparities

In the 100-word window analysis, Black racial terms accounted for 45.70% of all racial disease co-occurrences, despite Black people making up only 12.05% of the US population. Conversely, White racial terms appeared in only 37.66% of co-occurrences, despite representing 57.84% of the population. Gender disparities were also evident: male markers were associated with disease mentions more often (56.36%) than female markers (43.64%).

Comparison of disease co-occurrence proportions in training data versus actual US population proportions.

Bias Amplification and Source Variance

While the training data is biased, the study found that GPT-4 often amplifies these biases. For diseases like hypertension and sarcoidosis, the degree to which GPT-4 associates the condition with Black patients far outstrips the overrepresentation found in the training data. Furthermore, bias is not uniform; "general knowledge" sources like Wikipedia and Books had much higher rates of demographic associations than technical sources like GitHub or arXiv.

Conclusion

The researchers emphasize that simply removing demographic references is not the solution. Instead, they advocate for transparent reporting of training data composition and the development of targeted mitigation strategies. Understanding these "seeds of stereotypes" is a critical step toward preventing harmful biases from blooming into clinical AI applications.

Summary updated

Abstract

Background Advancements in Large Language Models (LLMs) hold transformative potential in healthcare, however, recent work has raised concern about the tendency of these models to produce outputs that display racial or gender biases. Although training data is a likely source of such biases, exploration of disease and demographic associations in text data at scale has been limited. Methods We conducted a large-scale textual analysis using a dataset comprising diverse web sources, including Arxiv, Wikipedia, and Common Crawl. The study analyzed the context in which various diseases are discussed alongside markers of race and gender. Given that LLMs are pre-trained on similar datasets, this approach allowed us to examine the potential biases that LLMs may learn and internalize. We compared these findings with actual demographic disease prevalence as well as GPT-4 outputs in order to evaluate the extent of bias representation. Results Our findings indicate that demographic terms are disproportionately associated with specific disease concepts in online texts. gender terms are prominently associated with disease concepts, while racial terms are much less frequently associated. We find widespread disparities in the associations of specific racial and gender terms with the 18 diseases analyzed. Most prominently, we see an overall significant overrepresentation of Black race mentions in comparison to population proportions. Conclusions Our results highlight the need for critical examination and transparent reporting of biases in LLM pretraining datasets. Our study suggests the need to develop mitigation strategies to counteract the influence of biased training data in LLMs, particularly in sensitive domains such as healthcare.
BibTeX
@misc{hansen2024seeds,
  author = {Lasse Hyldig Hansen and Nikolaj Andersen and Jack Gallifant and Liam G. McCoy and James K. Stone and Nura Izath and Marcela Aguirre-Jerez and Danielle S. Bitterman and Judy Gichoya and Leo Anthony Celi},
  title = {{Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources}},
  year = {2024},
  eprint = {2405.05049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.05049},
}
Figure