LLMs Reproduce Stereotypes of Sexual and Gender Minorities

Ruby Ostrow, Adam LopezEMNLP 2025

TL;DR

This paper investigates how large language models (LLMs) perpetuate stereotypes about sexual and gender minorities. By adapting a social psychology model (Stereotype Content Model) and applying it to both survey responses and text generation, the authors demonstrate that LLMs, like humans, exhibit biased perceptions, particularly towards nonbinary and bisexual individuals, and can amplify these harms in creative writing.

What Problem Does This Paper Address?

While there's a growing awareness of gender bias in NLP, much of the research has focused on a binary view of gender (men vs. women) and often conflates gender with sex, neglecting the complexities of sexual orientation and nonbinary identities. This paper aims to address this gap by examining biases in LLMs towards a broader spectrum of sexual and gender minorities. The authors are concerned about "representational harm," where LLMs perpetuate and amplify negative stereotypes, potentially leading to real-world harms. They seek to understand if LLMs mirror human societal stereotypes and if these biases manifest in realistic LLM applications like creative writing.

Methods

The core of the study is the application of the Stereotype Content Model (SCM), a well-established framework in social psychology. The SCM posits that stereotypes can be reduced to two dimensions: Warmth and Competence.

The researchers employed three main methods:

  1. SCM Survey (Rating, Selection, Elicitation):
    • Rating: Participants (both human and LLM) rated how descriptive certain attributes (Warmth/Competence terms) were of specific social groups.
  2. Selection: Participants chose the single most descriptive attribute from a list.
  3. Elicitation: Participants generated free-text keywords describing the groups. This allowed for a broader exploration beyond pre-defined SCM terms.
  4. Crucially, they conducted a new human survey to include groups not covered in prior SCM research (e.g., nonbinary, bisexual, lesbian).
  5. LLM Prompt Design: They adapted existing SCM survey prompts for LLMs, using three specific models: GPT 3.5-turbo, Gemini-1.5-flash, and LLaMA 2-7b-chat-hf. They also included prompts designed to elicit Chain-of-Thought reasoning.
  6. Text Generation Analysis: To assess more realistic use cases, they prompted LLMs to generate descriptions and stories about individuals from various gender and sexuality groups. They then analyzed the generated text for words associated with Warmth and Competence using semantic similarity and odds ratios.

Datasets

The paper does not explicitly mention training datasets for the LLMs, as they are using pre-trained, widely available models (GPT 3.5, Gemini 1.5, LLaMA 2).

For evaluation, they created a new dataset by collecting responses from:

  • Human Participants: 97 participants recruited via Prolific, filtered for English fluency.
  • LLM Responses: Outputs from the three tested LLMs across the survey and text generation tasks.

The elicited keywords from both humans and LLMs were analyzed using a pre-existing psychological measurement instrument that associates terms with dimensions like Morality/Sociability (linked to Warmth) and Agency/Ability (linked to Competence).

Key Insights and Ideas

  • LLMs mirror human stereotypes: The most striking finding is that LLMs, much like humans, exhibit similar patterns of stereotyping towards sexual and gender minorities. This suggests that biases present in human society are being learned and reproduced by these models.
  • Focus on minority groups reveals deeper bias: While men and women are stereotyped (men high on competence, women high on warmth), the biases are more pronounced and negative for nonbinary and bisexual individuals, who are often described with terms related to hardship, confusion, or marginalization.
  • Text generation amplifies harms: Beyond simply reflecting stereotypes, LLMs can amplify them in creative writing. The generated narratives often reinforce negative portrayals, particularly for minority groups, by emphasizing their struggles and marginalization.
  • Gemini's divergence: While all LLMs showed biases, Gemini exhibited a more distinct pattern, sometimes showing an inverse relationship between Warmth and Competence compared to human responses.
  • SCM as a useful tool: The study validates the SCM as a robust framework for measuring stereotypes in LLMs, extending its application to more nuanced identity groups and realistic generation tasks.

Key Results

  • Survey Tasks:
    • Rating: Humans and LLMs consistently rated women highest for Warmth and men highest for Competence. Nonbinary and bisexual individuals were rated lowest on both axes, with nonbinary people often perceived most negatively. LLM ratings were generally more positive than human ratings.
  • Selection: Results largely mirrored the rating task, with men favored for Competence and women for Warmth.
  • Elicitation: Keywords for women and men aligned with Warmth and Competence respectively. Minority groups elicited terms related to confusion, indecisiveness, and historical discrimination.
  • Text Generation:
    • Qualitative analysis revealed that LLMs generated descriptions reinforcing Warmth-Competence stereotypes. For example, women were associated with "healing others" and men with "strength and confidence."
    • Nonbinary individuals were often described with terms like "misunderstood" and "whispers," while bisexual individuals faced descriptions of being "condemned" or "a threat."
    • LLMs frequently generated words related to struggle for nonbinary, bisexual, and lesbian individuals, highlighting marginalization.

Limitations and Future Work

  • Scope of Groups: The study focused on specific groups (women, men, nonbinary, gay, lesbian, bisexual, heterosexual) and acknowledges that this set is not exhaustive and that identities are not discrete.
  • Language and Culture: The study was conducted in English and focused on attitudes towards groups in the English-speaking world. It does not claim how results might differ in other languages or cultural contexts.
  • Human Survey Bias: The human survey reflects societal perceptions at a specific time (summer 2024) and may be influenced by current political climates, whereas LLM training data is more temporally diverse.
  • SCM Reductionism: The SCM framework, while widely applicable, is inherently reductive and may ignore other important aspects of stereotypes.
  • Future Work: The authors suggest that LLM users should be aware of these biases and exercise caution. Future work could explore other bias measurement frameworks, different languages, and a wider range of LLMs. They also highlight the need for LLM vendors to address these issues.

Summary updated

BibTeX
@inproceedings{ostrow2025llms,
  author = {Ruby Ostrow and Adam Lopez},
  title = {{LLMs Reproduce Stereotypes of Sexual and Gender Minorities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure