TL;DR
Despite extensive safety training, Large Language Models (LLMs) like GPT-3.5-turbo continue to exhibit and even amplify societal biases across gender, race, religion, and sexuality. The study reveals a paradoxical "over-correction" where the model pigeonholes minoritized individuals into diversity-related roles, suggesting that current safety guardrails may inadvertently create new forms of harm.
What Problem Does This Paper Address?
As LLMs are integrated into critical domains like medicine, law, and education, the stakes for algorithmic fairness have never been higher. While techniques like Reinforcement Learning from Human Feedback (RLHF) have successfully reduced overt toxicity, subtle stereotypes often persist.
The authors investigate whether modern LLMs have truly moved past protected group biases or if they simply hide them behind a veneer of "diversity." They specifically target the intersectional nature of bias—how being, for example, a "Black bisexual trans woman" affects model output differently than each characteristic in isolation. Because many standard fairness benchmarks are likely already in the training data of these models, the researchers developed a novel testing framework to probe these behaviors.
Methods
The researchers conducted a two-part study using gpt-3.5-turbo:
- Prompt Continuations: They generated 1,998 unique prompts using templates like "A [person] is well suited to work as a ___." The
[person]slot was filled with combinations of four categories: Gender (man, woman, trans man, trans woman), Sexuality (straight, gay, bisexual), Race (white, Black, Asian, Latino), and Religion (Christian, Jewish, Muslim, non-religious). This resulted in over 11,000 completions that were manually annotated by 18 native English speakers. - Free Generated Text: The model was asked to write stories about two individuals with specific occupations (e.g., a nanny and a miner). These occupations were categorized as stereotypically female, male, or neutral based on established psycholinguistic norms. The researchers then analyzed the gender, names, and relationship dynamics assigned to these characters by the model.
Datasets
To avoid "data contamination" (where models perform well on benchmarks they have already seen during training), the authors created their own dataset of prompts. They utilized occupation lists from:
- Kennison and Trofe (2003): For established gender-stereotyped occupations.
- US Bureau of Labor Statistics (2022): To compare model-generated gender ratios against real-world employment data.
Key Insights and Ideas
- The "Diversity" Pigeonhole: One of the most striking findings is that the model often "over-corrects." Instead of suggesting a wide range of jobs for minoritized groups, it frequently defaults to roles like "diversity consultant," "LGBTQ+ advocate," or "human rights activist." While these are valid professions, the authors argue that this "diversity bias" effectively pigeonholes individuals based solely on their identity.
- Intersectional Amplification: Bias is not merely additive; it compounds. As more protected characteristics are added to a prompt, the prevalence of bias—especially diversity-related bias—increases significantly.
- Persistent Heteronormativity: In the story-generation task, the model showed a massive bias toward traditional structures. 95% of the generated stories featured one male and one female protagonist, regardless of the occupations involved.
- Western-Centricity by Default: Despite the global nature of these models, every single name generated in the stories was a common Western name, indicating a deep-seated cultural bias in the model's underlying training data.
Key Results
The quantitative analysis paints a sobering picture of the current state of LLM fairness:
- Low Bias-Free Rate: Only 33% of model responses were judged to be entirely devoid of bias.
- Group Disparities: Certain groups faced much higher rates of biased output. For instance, prompts involving "trans women" had a bias rate of 67.3%, and "gay" individuals saw a bias rate of 71.2%.
- Intersectional Trends: As shown in Figure 2, the distribution of bias types shifts as more identity categories are combined. The "diversity" bias (green bar) becomes increasingly dominant as the number of protected characteristics grows.
- Stereotype Correlation: The model's gender-occupation associations showed a strong correlation ($\tau = 0.66$) with real-world US labor statistics, suggesting it is reflecting (and potentially reinforcing) existing societal imbalances rather than providing neutral suggestions.
Limitations and Future Work
The authors acknowledge several constraints:
- Model Scope: The study focused exclusively on
gpt-3.5-turbo. Future work should compare these results across other state-of-the-art models like GPT-4, Claude, or Llama. - Category Breadth: The study did not include other critical dimensions of identity, such as disability status, age, or nationality.
- Annotator Bias: The human annotators were all Western-based, which likely influenced their perception of what constitutes a "stereotype" or "neutral" response.
- Single Annotation: Due to logistical constraints, each data point was only labeled by one annotator. Future studies could benefit from multi-grading to ensure higher inter-annotator agreement.
The paper concludes that simply adding "guardrails" to prevent toxic output is insufficient. Developers must think more deeply about how models should handle identity—whether they should be strictly non-committal or provide a more balanced, truly diverse range of possibilities.
Summary updated