MISGENDERED: Limits of Large Language Models in Understanding Pronouns

Tamanna Hossain, Sunipa Dev, Sameer Singh

Introduction

As Large Language Models (LLMs) are increasingly integrated into daily digital interactions, their ability to navigate social nuances—particularly gender identity—has become a critical area of AI safety and ethics. While previous research has largely focused on binary gender bias (male vs. female), the MISGENDERED framework reveals a significant failure in how these models handle non-binary gender expressions. Even when explicitly instructed on an individual's preferred pronouns, popular models frequently default to harmful biases, particularly when encountering neo-pronouns like xe/xem.

The Problem: Misgendering through Erasure

Most gender bias research in Natural Language Processing (NLP) operates within a binary paradigm. However, as gender-diverse identities become more visible, AI systems must respect a wider range of pronouns, including gender-neutral options (they/them) and various neo-pronouns (ze, xe, thon).

The core issue identified is misgendering through erasure. This occurs when a model, despite being provided with explicit context (e.g., "Aamari’s pronouns are xe/xem..."), ignores the instruction and defaults to a binary pronoun. This failure is more than a grammatical slip; it represents a breakdown in the model's ability to follow context and a perpetuation of real-world harm against marginalized communities.

The MISGENDERED Framework

To evaluate this phenomenon, researchers developed a unified approach to test both masked language models (e.g., BERT, T5) and auto-regressive models (e.g., GPT, LLaMA). The framework utilizes three primary components:

  1. Template-Based Prompting: The researchers created sentences with missing pronoun slots. Each instance begins with a "declaration" of the person's pronouns—either explicitly stated or provided parenthetically—followed by a sentence requiring the model to fill in the correct pronoun.
  2. Constrained Decoding: To ensure fair comparison across different architectures, the authors employed constrained decoding. Instead of allowing open-ended generation, they forced models to choose the most likely candidate from 11 specific pronoun groups (covering binary, neutral, and neo-pronouns).
  3. Zero-Shot and Few-Shot Evaluation: Models were tested both "out-of-the-box" and with "shots" (contextual examples) to determine if in-context learning could mitigate inherent biases.

Dataset and Corpus Analysis

The study introduced the MISGENDERED dataset, a massive benchmark consisting of 3.8 million instances. The dataset includes:

  • 11 Pronoun Sets: Two binary sets, one gender-neutral (they), and eight neo-pronoun sets.
  • 500 Names: Popular US names categorized as male, female, or unisex.
  • 50 Templates: Ten variations for each of the five pronoun forms (Nominative, Accusative, Possessive-Dependent, Possessive-Independent, and Reflexive).

Beyond the benchmark, the researchers analyzed the C4 and Pile datasets—standard training corpora for LLMs. Their analysis revealed a staggering data imbalance: masculine pronouns appear 220 times more frequently in the C4 dataset than the neo-pronoun xe. This scarcity suggests that models simply do not see enough examples of neo-pronouns to learn their grammatical functions effectively.

Key Insights: The Name-Gender Association Trap

A primary driver of model failure is the Name-Gender Association. LLMs have effectively "memorized" that certain names are linked to specific genders based on their training data. If a person named "Katherine" is identified as using xe/xem pronouns, the model is highly likely to ignore the prompt's instruction and use she/her because the name-based association is stronger than the contextual instruction.

Furthermore, the study found that scaling is not a panacea. While some models like OPT showed improvement as they grew in size, others like BLOOM showed a negative correlation with size regarding neutral pronouns. This suggests that larger models may become more "confident" in the biases they have internalized from their training data rather than becoming more adaptable to context.

Results and Performance Gaps

The performance gap between binary and non-binary pronoun usage is stark:

  • Zero-Shot Accuracy: While models achieved approximately 75.9% accuracy for binary pronouns, accuracy dropped to 34.2% for gender-neutral (they/them) pronouns and a mere 7.7% for neo-pronouns.
  • Few-Shot Adaptation: Providing 20 examples in the prompt (20-shot) significantly improved neo-pronoun accuracy to 64.7% for LLaMA-7B. While this demonstrates the power of in-context learning, it still falls short of binary performance levels.
  • Declaration Style: The researchers found that "parenthetical" declarations (e.g., "Aamari (they/them) is...") were more effective for binary and neutral pronouns, whereas "explicit" declarations (e.g., "Aamari's pronouns are...") were necessary for models to better process neo-pronouns.

Conclusion

The MISGENDERED framework highlights a significant technical and ethical hurdle for LLMs. The reliance on name-based heuristics and the extreme scarcity of diverse gender expressions in training data lead to systemic misgendering. While few-shot prompting offers a partial remedy, the authors argue for more fundamental changes, including the development of better debiasing techniques and the curation of more inclusive pre-training datasets. By releasing their code and dataset, the researchers aim to move the AI community beyond the binary paradigm toward more equitable language technologies.

Summary updated

BibTeX
@inproceedings{hossain2023misgendered,
  author = {Tamanna Hossain and Sunipa Dev and Sameer Singh},
  title = {{MISGENDERED: Limits of Large Language Models in Understanding Pronouns}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.293},
}
Figure