Abstract
Targeted evaluations have found that machine translation systems often output incorrect gender, even when the gender is clear from context. Furthermore, these incorrectly gendered translations have the potential to reflect or amplify social biases. We propose a gender-filtered self-training technique to improve gender translation accuracy on unambiguously gendered inputs. This approach uses a source monolingual corpus and an initial model to generate gender-specific pseudo-parallel corpora which are then added to the training data. We filter the gender-specific corpora on the source and target sides to ensure that sentence pairs contain and correctly translate the specified gender. We evaluate our approach on translation from English into five languages, finding that our models improve gender translation accuracy without any cost to generic translation quality. In addition, we show the viability of our approach on several settings, including re-training from scratch, fine-tuning, controlling the balance of the training data, forward translation, and back-translation.
Introduction
Machine translation (MT) systems frequently struggle when translating from a language without grammatical gender, such as English, into languages with extensive gender markings like German, Italian, or Russian. A common failure mode is the reliance on "selection bias" or stereotypes. For instance, an English sentence like "I am a doctor" provides no explicit gender cue, but a German translation must choose between the masculine Arzt and the feminine Ärztin.
Existing systems often default to masculine forms because training data is typically sourced from the web, where masculine-referring sentences significantly outnumber feminine ones. This paper addresses the challenge of improving accuracy on unambiguously gendered inputs—cases where context, such as a coreferential pronoun, clearly dictates the correct gender choice.
Methodology: Gender-Filtered Self-Training
The researchers developed a pipeline designed to balance the training data through high-precision self-training. The process begins with a baseline Neural Machine Translation (NMT) model trained on standard parallel data.
As shown in the figure above, the pipeline follows these key steps:
- Source Filtering: A massive monolingual English corpus is scanned for sentences that are clearly masculine or feminine based on a curated list of 104 word pairs (e.g., brother/sister) and pronouns.
- Forward Translation: The baseline model translates these gender-specific sentences into the target language.
- Target Morphological Filtering: This critical step uses a morphological tagger on the generated translations to verify that the gender was preserved. If an English sentence about a "daughter" results in a masculine noun in the target language, the pair is discarded to prevent error propagation.
- Data Augmentation: The resulting high-confidence, gender-balanced "pseudo-parallel" pairs are added to the original training set.
- Final Training: A new model is trained on this augmented dataset.
The Data Imbalance Problem
A primary driver of gender bias in MT is the scarcity of feminine examples in standard corpora. The authors' analysis of training sets for five language pairs revealed a stark disparity.
As illustrated in the table, feminine-specific sentences (Fem) make up as little as 0.5% to 1.9% of the total data, while masculine sentences (Msc) are significantly more prevalent. By identifying and augmenting the training set with filtered feminine examples, the researchers were able to provide the model with the necessary signals to learn correct gender mappings.
Key Results and Insights
The proposed model, denoted as ST$_{Gender}$, yielded significant improvements across several benchmarks:
- Accuracy Gains: On the WinoMT benchmark, average gender accuracy improved from 52.4% to 58.8%. The most substantial jumps occurred in English-to-German and English-to-Italian, with improvements of approximately 10-11 percentage points.
- Bias Reduction: The method effectively reduced the "gender gap" ($\Delta R$). In English-to-German, the directional bias dropped from 18.8 to -0.3, indicating nearly balanced performance between genders.
- Preserving Generic Quality: Crucially, these gains did not come at the expense of general translation quality. BLEU scores remained stable or improved slightly, demonstrating that targeted bias mitigation can be integrated without degrading the system's overall utility.
- Filtering vs. Quantity: The aggressive target-side filtering often discarded over 75% of generated translations. However, this precision-oriented approach was found to be superior to simply adding more (potentially incorrect) data, as it prevents the model from reinforcing its own biases.
Conclusion and Future Work
Filtered self-training offers a robust, low-cost method for improving gender accuracy in machine translation. While the current study focused on binary gender due to existing benchmark limitations, it provides a framework for addressing data imbalances in NMT. Future research may explore extending these techniques to non-binary gender identities and applying similar filtering methods to back-translation or synthetic data generation to further refine model performance in low-resource settings.
Summary updated