SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models

Manish Nagireddy, Lamogha Chiazor, Moninder Singh, Ioana Baldini

Abstract

Current datasets for unwanted social bias auditing are limited to studying protected demographic features such as race and gender. In this work, we introduce a comprehensive benchmark that is meant to capture the amplification of social bias, via stigmas, in generative language models. Taking inspiration from social science research, we start with a documented list of 93 US-centric stigmas and curate a question-answering (QA) dataset which involves simple social situations. Our benchmark, SocialStigmaQA, contains roughly 10K prompts, with a variety of prompt styles, carefully constructed to systematically test for both social bias and model robustness. We present results for SocialStigmaQA with two open source generative language models and we find that the proportion of socially biased output ranges from 45% to 59% across a variety of decoding strategies and prompting styles. We demonstrate that the deliberate design of the templates in our benchmark (e.g., adding biasing text to the prompt or using different verbs that change the answer that indicates bias) impacts the model tendencies to generate socially biased output. Additionally, through manual evaluation, we discover problematic patterns in the generated chain-of-thought output that range from subtle bias to lack of reasoning. Warning: This paper contains examples of text which are toxic, biased, and potentially harmful.

Introduction

While the AI community has made significant strides in auditing models for biases related to "protected attributes" such as race, gender, and religion, many other forms of social discrimination remain under-studied. This research targets social stigmas: devalued attributes that reduce a person "from a whole and usual person to a tainted, discounted one." Examples include being voluntarily childless, having facial scars, living in a trailer park, or being a former sex worker. Because these stigmas are often more nuanced than demographic categories, they are frequently overlooked in standard safety evaluations. SocialStigmaQA addresses the gap between a model appearing "fair" on standard benchmarks and its actual behavior in complex, everyday social situations where stigmas might trigger discriminatory output.

The SocialStigmaQA Benchmark

The researchers developed a framework to systematically "probe" models using a Question-Answering (QA) format. The benchmark consists of 10,360 prompts covering 93 US-centric stigmas derived from social science research.

Template and Prompt Design

The dataset utilizes 37 templates representing mundane social situations, such as hiring a candidate or renewing a tenant's lease. To test model robustness, four prompt styles were implemented for every situation:

  • Original: A neutral description of the situation involving a person with a stigma.
  • Positive Bias: Adds text nudging the model toward an unbiased answer (e.g., "the candidate is a perfect fit").
  • Doubt Bias: Adds text expressing uncertainty ("I don't know what to do").
  • No-Stigma: A control version with no mention of a stigma to establish a baseline.
Table 1: Examples of the four prompt styles used in SocialStigmaQA to evaluate model robustness and bias.

Evaluation Setup

The benchmark was tested on two prominent open-source models: Flan-T5-XXL (11B parameters) and Flan-UL2 (20B parameters). The researchers employed two decoding strategies: Greedy Decoding and Nucleus Sampling with Chain-of-Thought (CoT). Evaluation involved automated string matching for final "yes/no" answers and manual annotation for CoT reasoning paths.

Experimental Results

Across various settings, the proportion of socially biased outputs ranged from 45% to 59%.

Table 2: Bias proportions for Flan-T5 and Flan-UL2 across different decoding methods.

Key Findings

  • Model Comparison: Flan-UL2, the larger model, generally exhibited more bias than Flan-T5-XXL, particularly under greedy decoding (59.2% vs 45.0%).
  • The CoT Paradox: While Chain-of-Thought (CoT) is intended to improve reasoning, it proved to be a double-edged sword. In some cases, CoT increased the proportion of biased answers. Even when the final answer was unbiased, the reasoning path often contained subtle justifications that revealed underlying stigmas.
  • Stigma Hierarchy: Models showed varying levels of bias across categories. The highest levels were found in categories related to sexual behavior (e.g., sex offenders, HIV status) and drug consumption. Interestingly, stigmas directly mentioning race or gender often showed less bias, likely due to intensive safety-tuning in those specific areas.
  • Brittleness of Safety Tuning: Models were found to be highly sensitive to phrasing. Adding a single word like "however" or changing the random seed could swing bias scores by over 10 points, suggesting that current safety alignment is "brittle" rather than deeply rooted in social understanding.

Discussion and Future Work

The SocialStigmaQA benchmark highlights that current generative models are prone to amplifying social stigmas. However, the study has limitations, notably its US-centric focus. Stigmas vary significantly across different cultures and languages. Furthermore, some stigmas involve illegal activities, where a model's response might be interpreted as "logical" or "legalistic," complicating the definition of bias.

Future work includes expanding the dataset to cover more global axes of discrimination and using the labeled output from SocialStigmaQA to fine-tune models via Reinforcement Learning from Human Feedback (RLHF) to better align them with social fairness goals.

Summary updated

Abstract

Current datasets for unwanted social bias auditing are limited to studying protected demographic features such as race and gender. In this work, we introduce a comprehensive benchmark that is meant to capture the amplification of social bias, via stigmas, in generative language models. Taking inspiration from social science research, we start with a documented list of 93 US-centric stigmas and curate a question-answering (QA) dataset which involves simple social situations. Our benchmark, SocialStigmaQA, contains roughly 10K prompts, with a variety of prompt styles, carefully constructed to systematically test for both social bias and model robustness. We present results for SocialStigmaQA with two open source generative language models and we find that the proportion of socially biased output ranges from 45% to 59% across a variety of decoding strategies and prompting styles. We demonstrate that the deliberate design of the templates in our benchmark (e.g., adding biasing text to the prompt or using different verbs that change the answer that indicates bias) impacts the model tendencies to generate socially biased output. Additionally, through manual evaluation, we discover problematic patterns in the generated chain-of-thought output that range from subtle bias to lack of reasoning. Warning: This paper contains examples of text which are toxic, biased, and potentially harmful.
BibTeX
@inproceedings{nagireddy2024socialstigmaqa,
  author = {Manish Nagireddy and Lamogha Chiazor and Moninder Singh and Ioana Baldini},
  title = {{SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=jvLfYhOzGh},
}
Figure