Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

Ana Valeria González-Garduño, Maria Barrett, Rasmus Hvingelby, Kellie Webster, Anders Søgaard

Abstract

The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages. English challenge datasets such as GAP and WinoGender highlight model preferences that are "hallucinatory"—for instance, disambiguating gender-ambiguous occurrences of 'doctor' as male. We show that for languages with Type B reflexivization (e.g., Swedish, Russian, Danish, and Chinese), we can construct multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions. In these languages, the coreferential reading requires a non-gendered reflexive pronoun, while gendered possessive pronouns are strictly anti-reflexive. We present the Anti-reflexive Bias Challenge (ABC), a multilingual, multi-task dataset spanning four languages and four NLP tasks. We find evidence for gender bias across all task-language combinations and correlate model bias with national labor market statistics.

Introduction

Most existing benchmarks for measuring gender bias in NLP focus exclusively on English. In English, pronouns are often ambiguous. For example, in the sentence "The surgeon put a book on his table," the word "his" could refer to the surgeon or another male individual. If an NLP model assumes "his" refers to the surgeon, it is making a biased "hallucination" based on occupational stereotypes, but it is not technically violating English grammar.

This focus on English misses a crucial opportunity. In languages with Type B reflexivization, this ambiguity does not exist. These languages have distinct pronouns for coreferential readings (reflexive) and disjoint readings (anti-reflexive). If a model links an anti-reflexive pronoun to the subject based on a stereotype, it is making a hard grammatical error rather than a semantic assumption.

Linguistic Background: Type B Reflexivization

The core of the ABC dataset lies in the structural difference between Type A and Type B languages. In Type A languages like English, the same pronoun can often serve both reflexive and non-reflexive functions in possessive contexts. In contrast, Type B languages utilize a specific reflexive pronoun for the third person to indicate coreference with the subject.

Comparison of Type A and Type B reflexivization systems.

For example, in Danish, using the gender-neutral reflexive pronoun sine strictly means the object belongs to the subject. Using the masculine anti-reflexive pronoun hans strictly means it belongs to another person. If a model predicts hans refers to the subject "surgeon" simply because surgeons are stereotypically male, it has committed an unambiguous morpho-syntactic error.

The Anti-reflexive Bias Challenge (ABC)

To evaluate this phenomenon, we created the Anti-reflexive Bias Challenge (ABC). We developed hand-written templates using 60 occupations with known gender distributions (e.g., mechanic, receptionist) and generated 4,560 base sentences. These were translated into Danish (da), Russian (ru), Swedish (sv), and Chinese (zh).

We adapted these templates to evaluate models across four distinct NLP tasks:

  1. Natural Language Inference (NLI): Testing if models wrongly predict "entailment" between a premise with a reflexive pronoun and a hypothesis with an anti-reflexive pronoun.
  2. Machine Translation (MT): Observing if models default to reflexive or anti-reflexive translations based on the gender of the source English pronoun.
  3. Coreference Resolution: Testing if models erroneously cluster an anti-reflexive pronoun with the sentence subject.
  4. Language Modeling (LM): Comparing model perplexity when forced to predict masculine vs. feminine anti-reflexives in place of a correct reflexive pronoun.

Experimental Results

Our results demonstrate that gender bias is powerful enough to override strict grammatical rules across all tested languages and tasks. Models consistently performed worse on the ABC dataset than on standard benchmarks.

Model performance on the ABC dataset across tasks and languages.

Key Findings

  • NLI Performance Drop: In Chinese (zh), the XLM-15 model achieved 0.742 accuracy on the standard XNLI test set but plummeted to 0.330 on ABC, frequently over-predicting entailment for anti-reflexive pronouns.
  • Translation Bias: Machine Translation systems heavily favored translating masculine English pronouns to reflexives. In Danish, 93.7% of masculine pronouns were translated into reflexives, compared to only 72.9% of feminine pronouns.
  • Coreference Errors: The Chinese coreference model erroneously clustered masculine anti-reflexive pronouns with the subject 17% of the time, showing a clear preference for masculine subjects in coreferential roles even when grammatically forbidden.
  • Labor Statistic Correlation: Across all tasks, model errors on sentences with feminine pronouns showed a strong, statistically significant negative correlation with the actual percentage of women in those occupations according to national labor statistics.

Conclusion

The ABC dataset proves that modern NLP models are so heavily influenced by occupational gender stereotypes that they will break fundamental grammatical rules to satisfy semantic biases. By moving beyond English "hallucinations" to unambiguous grammatical errors in Type B languages, we provide a more rigorous testbed for evaluating the depth of gender bias in multilingual systems. Our findings also suggest that model biases correlate more closely with localized national labor statistics than with US-centric data, highlighting the need for culturally aware bias evaluation.

Summary updated

Abstract

The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and WinoGender highlight model preferences that are "hallucinatory", e.g., disambiguating gender-ambiguous occurrences of 'doctor' as male doctors. We show that for languages with type B reflexivization, e.g., Swedish and Russian, we can construct multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions: In these languages, the direct translation of 'the doctor removed his mask' is not ambiguous between a coreferential reading and a disjoint reading. Instead, the coreferential reading requires a non-gendered pronoun, and the gendered, possessive pronouns are anti-reflexive. We present a multilingual, multi-task challenge dataset, which spans four languages and four NLP tasks and focuses only on this phenomenon. We find evidence for gender bias across all task-language combinations and correlate model bias with national labor market statistics.
BibTeX
@inproceedings{gonzalezgarduno2020type,
  author = {Ana Valeria González-Garduño and Maria Barrett and Rasmus Hvingelby and Kellie Webster and Anders Søgaard},
  title = {{Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=11tiVmcu9MT},
}
Figure