Abstract
Current datasets for unwanted social bias auditing are limited to studying protected demographic features such as race and gender. In this work, we introduce a comprehensive benchmark that is meant to capture the amplification of social bias, via stigmas, in generative language models. Taking inspiration from social science research, we start with a documented list of 93 US-centric stigmas and curate a question-answering (QA) dataset which involves simple social situations. Our benchmark, SocialStigmaQA, contains roughly 10K prompts, with a variety of prompt styles, carefully constructed to systematically test for both social bias and model robustness. We present results for SocialStigmaQA with two open source generative language models and we find that the proportion of socially biased output ranges from 45% to 59% across a variety of decoding strategies and prompting styles. We demonstrate that the deliberate design of the templates in our benchmark (e.g., adding biasing text to the prompt or using different verbs that change the answer that indicates bias) impacts the model tendencies to generate socially biased output. Additionally, through manual evaluation, we discover problematic patterns in the generated chain-of-thought output that range from subtle bias to lack of reasoning. Warning: This paper contains examples of text which are toxic, biased, and potentially harmful.
Introduction
While the AI community has focused significant effort on auditing Large Language Models (LLMs) for biases related to "protected attributes" like race and gender, many other forms of social prejudice remain under-studied. Social science identifies dozens of stigmas—devalued attributes like having a criminal record, being voluntarily childless, or living in a trailer park—that lead to real-world discrimination. As LLMs are increasingly deployed in sensitive contexts such as job screening or medical advice, it is critical to determine if they amplify these stigmas.
Methodology
To address this gap, researchers developed SocialStigmaQA, a benchmark containing 10,360 prompts. The dataset was constructed using 37 social situation templates (e.g., "Should I hire this person?") injected with 93 different US-centric stigmas covering health, social status, and personal choices.
To test model robustness and the impact of framing, the researchers utilized four distinct prompt styles:
- Original: A neutral mention of the stigma.
- Positive Bias: Nudging the model toward an unbiased answer (e.g., "The candidate is a perfect fit...").
- Doubt Bias: Adding a layer of uncertainty (e.g., "I'm not sure what to do...").
- No-Stigma: A control group to establish baseline behavior.
The benchmark was evaluated on two open-source models: Flan-T5-XXL (11B parameters) and Flan-UL2 (20B parameters). The study compared "Greedy" decoding with "Nucleus Sampling" using Chain-of-Thought (CoT) prompting to observe the models' internal reasoning processes.
Key Results and Analysis
The evaluation revealed that both models frequently generate biased outputs, with bias proportions ranging from 45% to 59% depending on the decoding strategy and model size.
Key Insights:
- The CoT "Truth Serum": Chain-of-Thought prompting often surfaced "subtle bias." A model might provide an unbiased final answer but justify it with problematic logic (e.g., "Even though they have a criminal record and might be dangerous, yes, hire them").
- Fragility of Nudges: While "positive bias" text reduced biased answers, it did not eliminate them. Models still discriminated in roughly half of the cases even when explicitly told a candidate was a "perfect fit."
- Stigma Hierarchy: Bias was not uniform. Models showed the highest prejudice toward stigmas related to sexual behavior and drug consumption, while showing less bias toward race or gender, likely due to more intensive safety alignment in those areas.
- Inconsistency: Model behavior proved highly sensitive to the "random seed," with bias scores swinging by more than 10 points, suggesting significant instability in model safety.
Conclusion
SocialStigmaQA demonstrates that current LLMs significantly amplify social stigmas. While the current dataset is US-centric and English-only, it provides a framework for the community to expand auditing to more axes of discrimination. The authors suggest that this benchmark can be used to fine-tune models to "unlearn" specific social prejudices through reward-based methods.
Summary updated