Introduction
As Large Language Models (LLMs) like GPT-4 are increasingly integrated into healthcare for tasks such as clinical summarization and diagnosis, researchers have noted a disturbing trend: these models often produce outputs reflecting deep-seated racial and gender stereotypes. This paper seeks to quantify the "representational harms" within the massive, uncurated datasets used to train these models. By analyzing 1.2 trillion tokens of web-based data, the authors aim to determine if the biases seen in AI outputs are reflections of the internet's "collective consciousness" or if the models are amplifying existing disparities.
Methodology
The authors employed a large-scale textual co-occurrence analysis to measure how often specific diseases are mentioned alongside demographic markers. The process involved three key steps:
- Keyword Definition: Curating a dictionary for 18 specific diseases and various racial/gender groups using medical (MeSH) headings.
- Proximity Mapping: Using "token windows" (20, 100, 200, and 500 words) to distinguish between passing mentions and direct contextual associations.
- Comparative Baseline: Comparing text proportions against actual US population disease prevalence and documented GPT-4 biases.
The primary dataset was RedPajama-Data-1T, an open-source collection designed to replicate the training data for Meta’s LLaMA-1, encompassing web scrapes (C4, Common Crawl), scientific papers (arXiv), books, and Wikipedia.
Key Findings
The "White Default" and Racial Skew
A striking finding was the "white defaultism" in English text—the tendency to assume a subject is White unless specified otherwise. This leads to a skewed data profile where "Black" is used as a specific marker, while "White" remains invisible. Consequently, Black individuals are significantly overrepresented in discussions of nearly all analyzed diseases compared to real-world population statistics.
Quantitative Disparities
In the 100-word window analysis, Black racial terms accounted for 45.70% of all racial disease co-occurrences, despite Black people making up only 12.05% of the US population. Conversely, White racial terms appeared in only 37.66% of co-occurrences, despite representing 57.84% of the population. Gender disparities were also evident: male markers were associated with disease mentions more often (56.36%) than female markers (43.64%).
Bias Amplification and Source Variance
While the training data is biased, the study found that GPT-4 often amplifies these biases. For diseases like hypertension and sarcoidosis, the degree to which GPT-4 associates the condition with Black patients far outstrips the overrepresentation found in the training data. Furthermore, bias is not uniform; "general knowledge" sources like Wikipedia and Books had much higher rates of demographic associations than technical sources like GitHub or arXiv.
Conclusion
The researchers emphasize that simply removing demographic references is not the solution. Instead, they advocate for transparent reporting of training data composition and the development of targeted mitigation strategies. Understanding these "seeds of stereotypes" is a critical step toward preventing harmful biases from blooming into clinical AI applications.
Summary updated