GFST: Gender-Filtered Self-Training for More Accurate Gender in Translation

Prafulla Kumar Choubey, Anna Currey, Prashant Mathur, Georgiana DinuEMNLP 2021 · OpenReview

Abstract

Neural Machine Translation (NMT) systems frequently exhibit a "masculine-by-default" bias, often translating gender-neutral or even gender-unambiguous source text into masculine target forms. This paper introduces Gender-Filtered Self-Training (GFST), a novel data augmentation technique designed to mitigate this bias. By leveraging large monolingual corpora and a rigorous target-side morphological filtering loop, GFST generates high-quality, gender-balanced pseudo-parallel data. Experimental results across five language pairs demonstrate that GFST significantly improves gender translation accuracy—achieving up to an 11.2-point gain—without sacrificing overall translation quality (BLEU).

Introduction

Machine translation systems often struggle with gender, frequently defaulting to masculine or gender-stereotypical outputs. For example, the English sentence "I am a doctor" spoken by a woman might be translated into German using the masculine form of "doctor" (Arzt) instead of the feminine (Ärztin).

This phenomenon stems primarily from sample bias: the parallel corpora used to train these models are heavily skewed, containing significantly more masculine-referring sentences than feminine ones. Even when the gender is unambiguous in the source text (e.g., "She is a doctor"), models may ignore these cues in favor of learned statistical biases. This research focuses specifically on improving accuracy for these unambiguous cases, where there is only one grammatically correct gender in the target language.

Methodology

The core of the proposed approach is Gender-Filtered Self-Training (GFST). Unlike standard self-training, which can reinforce existing biases by having a model learn from its own biased predictions, GFST introduces a verification step to ensure data quality.

The GFST workflow: from monolingual data to a gender-aware final model.

The workflow consists of four main stages:

  1. Initial Training: A baseline NMT model is trained on standard, often biased, parallel data.
  2. Source Filtering: A large English monolingual corpus is filtered to extract sentences that are clearly masculine or feminine based on pronouns (he/she) and gendered word lists (e.g., actor/actress).
  3. Forward Translation & Target Filtering: The baseline model translates these filtered sentences. Crucially, the authors apply Target Morphological Filtering. Using a morphological analyzer, they verify if the translation preserved the gender of the source. If a feminine source sentence results in a masculine target word, the pair is discarded. This prevents the propagation of biased errors into the training set.
  4. Final Training: The original parallel data is augmented with this verified, gender-balanced "pseudo-parallel" data. The model is then either retrained or fine-tuned.

Experimental Setup

The researchers evaluated GFST across five language pairs: English to German (en-de), French (en-fr), Hebrew (en-he), Italian (en-it), and Russian (en-ru). Training data was sourced from standard benchmarks like WMT and IWSLT, while the English News Crawl (2015–2017) provided the monolingual source for augmentation. Evaluation was conducted using professional-standard benchmarks, including WinoMT (for gender bias) and MuST-SHE (for gender translation accuracy).

Key Results

GFST consistently outperformed baseline models across the tested languages. On the WinoMT benchmark, average accuracy increased from 52.4% to 58.8%. The most dramatic improvements were observed in the English-to-German model, where accuracy rose from 75.5% to 85.4%.

Comparison of gender translation accuracy between the baseline and GFST models for select language pairs.

Key findings include:

  • Feminine Input Improvement: The largest gains were seen on feminine-gendered inputs, which are typically the most underserved by standard models.
  • Stability of General Quality: General translation quality, measured by BLEU scores, remained stable or improved slightly. This indicates that fixing gender bias does not require a trade-off in overall model performance.
  • Human Validation: Human evaluators confirmed the automatic metrics, noting significant improvements in gender accuracy for German and Italian.

Discussion and Insights

The success of GFST highlights the importance of Target Morphological Filtering. Without this "sanity check," self-training would simply amplify the model's existing biases. The authors also introduced a new metric, $\\Delta R$ (Delta Recall), to measure the difference in recall between masculine and feminine samples, providing a cleaner indicator of bias than traditional metrics.

Furthermore, the study demonstrates that fine-tuning an existing model on GFST data is a viable, low-cost alternative to retraining from scratch. While the current method focuses on binary gender due to the grammatical constraints of the target languages, it provides a robust framework for addressing data-driven biases in NMT.

Limitations and Future Work

The method relies on high-quality morphological analyzers, which may not be available for low-resource languages. Additionally, the filtering process is highly aggressive; in some instances, up to 98% of generated translations are discarded to ensure quality. Future research may explore synthetic data generation, such as gender-swapping source sentences, to further increase the volume of high-quality training data.

Summary updated

BibTeX
@inproceedings{choubey2021gfst,
  author = {Prafulla Kumar Choubey and Anna Currey and Prashant Mathur and Georgiana Dinu},
  title = {{GFST: Gender-Filtered Self-Training for More Accurate Gender in Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=8CMDOx5axiy},
}
Figure