Introduction
Machine translation (MT) systems face a significant challenge when translating from a language with minimal gender marking, such as English, to languages with rich grammatical gender, such as Spanish, French, or Russian. In these scenarios, a neutral English term like "doctor" must be assigned a gender in the target language (e.g., el médico vs. la médica). Without explicit context, models often default to societal stereotypes, favoring masculine forms for prestigious professions and feminine forms for caregiving roles.
While previous research relied on hand-crafted, "synthetic" sentences (e.g., "The doctor is here"), this study introduces a more robust, automated pipeline to detect these biases in the messy, complex language of the real world.
Methodology: Stress-Testing with Perturbations
The researchers developed a multi-step pipeline to identify "at-risk" gender pairs—sentences where a single word change causes the translation model to flip the gender assignment.
- Data Mining: The team started with nearly 30,000 naturally occurring English sentences mined from the r/Career subreddit.
- Perturbation via BERT: Instead of using static word lists, the pipeline uses BERT as a masked language model. By masking a human entity in a sentence, BERT suggests contextually appropriate alternatives. For example, in a sentence about a hospital, BERT might suggest replacing "doctor" with "nurse" or "surgeon."
- Translation and Analysis: These "minimal pairs" are translated into four target languages: French, German, Spanish, and Russian. A morphological analyzer then determines the gender of the translated noun. A pair is considered "at risk" if the two neutral English nouns result in different genders in the target language despite the identical sentence structure.
Key Results: The Masculine Prior
The pipeline generated over 40,000 perturbed pairs, revealing a massive "masculine prior" across all tested languages. In sentences where the model was consistent, the ratio of masculine to feminine translations was often hundreds to one.
As shown in the data, terms like "manager" or "employee" are overwhelmingly translated as masculine. Conversely, the few terms that consistently triggered feminine translations—such as "nurse," "secretary," and "babysitter"—align perfectly with societal stereotypes and labor statistics. This confirms that MT models are internalizing and amplifying real-world biases from their training data.
Insights and Future Directions
By moving beyond fixed word lists, the researchers discovered biased terms that standard benchmarks often miss, such as "fighter," which carries a strong masculine bias in MT encodings.
However, the study also identified technical hurdles, such as "fixed grammatical gender." In French, for instance, the word victime (victim) is always feminine regardless of the person's actual gender. Distinguishing these linguistic quirks from model bias currently requires manual filtering. Future work aims to expand this methodology to include adjectives and verbs, which also carry gender markings, and to explore bias when translating between two gender-marking languages.
Summary updated