Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

Ana Valeria González-Garduño, Maria Barrett, Rasmus Hvingelby, Kellie Webster, Anders Søgaard

Abstract

Ana Valeria González, Maria Barrett, Rasmus Hvingelby, Kellie Webster, Anders Søgaard. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Most research on gender bias in Natural Language Processing (NLP) has focused on English, where gendered pronouns are often ambiguous. In a sentence like "The doctor removed his mask," the pronoun "his" could refer to the doctor or a third party. When a model assumes the doctor is male, it is performing an unwarranted disambiguation, often termed a "hallucination."

However, many languages possess grammatical structures that remove this ambiguity. This research introduces the Anti-reflexive Bias Challenge (ABC), a multilingual dataset leveraging Type B reflexivization to pinpoint gender bias. In languages like Danish, Swedish, Russian, and Chinese, specific reflexive possessive pronouns (meaning "his/her own") exist alongside anti-reflexive gendered pronouns (meaning "someone else's").

Classification of reflexivization types across persons. Type B languages exhibit a gap in 3rd person reflexive marking, necessitating distinct anti-reflexive pronouns.

In these languages, if a model uses a gendered anti-reflexive pronoun to refer back to the subject, it is not merely making a biased guess; it is committing a hard grammatical error. This provides a unique testbed where gender bias leads to unambiguously incorrect predictions.

Methodology

The authors developed the ABC dataset using a template-based approach across four languages: Danish (da), Russian (ru), Swedish (sv), and Chinese (zh).

Dataset Construction

  1. Templates: 38 hand-crafted templates (e.g., "[OCCUPATION] lost [PRON] wallet") were translated and adapted for each language.
  2. Occupations: Templates were populated with 60 occupations (e.g., surgeon, nurse, engineer) sourced from prior bias literature.
  3. Labor Statistics: To evaluate the source of bias, the researchers correlated model errors with real-world workforce gender distributions from the U.S. Bureau of Labor Statistics (2015) and national statistics from the four target countries.

Multi-Task Evaluation

The study evaluated state-of-the-art models (including BERT, XLM, and Google Translate) across four distinct NLP tasks:

  • Natural Language Inference (NLI): Testing if models correctly identify that an anti-reflexive pronoun implies a different individual than the subject.
  • Machine Translation (MT): Observing if models incorrectly translate English possessives into reflexive pronouns to satisfy gender stereotypes.
  • Coreference Resolution: Checking if models wrongly link a gendered anti-reflexive pronoun to the subject.
  • Language Modeling (LM): Measuring if models assign higher probability (lower perplexity) to grammatically incorrect but stereotypically "consistent" sentences.

Key Results

The findings demonstrate that gender bias is a robust, cross-linguistic phenomenon that frequently overpowers grammatical constraints.

Performance metrics across NLI, MT, Coreference, and Language Modeling tasks, showing significant bias across all tested languages.

Task-Specific Findings

  • NLI Failures: XLM models consistently over-predicted entailment for anti-reflexive pronouns. They struggled to recognize that in a sentence like "The surgeon took his [anti-reflexive] book," the book does not belong to the surgeon.
  • Translation Bias: Models showed a heavy masculine default. In Russian, 69.3% of masculine pronouns were translated into reflexives (referring to the subject), while only 18.1% of feminine pronouns were treated similarly.
  • Coreference: In Chinese, models exhibited a higher error rate for masculine anti-reflexive pronouns (17%) compared to feminine ones (14.6%), wrongly linking them to the subject when the occupation was stereotypically male.
  • Language Modeling: BERT models showed lower perplexity for sentences that were grammatically incorrect but aligned with occupational gender stereotypes, suggesting the model "prefers" stereotypical errors over grammatical accuracy.

Discussion and Insights

Bias vs. Grammar

The most striking insight is that models will frequently violate hard linguistic rules to satisfy a gender stereotype. This suggests that the internal representations of gender in these models are sufficiently strong to override the learned rules of grammar.

The Influence of National Statistics

Model bias does not merely reflect generic internet data; it correlates significantly with real-world labor market statistics. Interestingly, national labor statistics were often better predictors of model bias than U.S. statistics. This suggests that multilingual models pick up on specific cultural and economic nuances present in the training data of their respective languages.

Conclusion and Future Work

The ABC dataset serves as a rigorous diagnostic tool for identifying gender bias that manifests as unambiguous grammatical failure. While this study focused on four languages, the authors note that Type B reflexivization is present in many other language families, including Slavic and Sino-Tibetan groups, offering a path for expanded benchmarking. Future work should utilize the ABC dataset to evaluate the efficacy of debiasing techniques in fixing these fundamental linguistic errors.

Summary updated

BibTeX
@inproceedings{gonzalezgarduno2020type,
  author = {Ana Valeria González-Garduño and Maria Barrett and Rasmus Hvingelby and Kellie Webster and Anders Søgaard},
  title = {{Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=11tiVmcu9MT},
}
Figure