MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

Anna Currey, Maria Nadejde, Raghavendra Reddy Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, Georgiana Dinu

Abstract

Neural Machine Translation (NMT) systems frequently struggle with gender accuracy, often defaulting to societal stereotypes or failing to maintain consistency across sentences. We present MT-GenEval, a realistic benchmark designed to evaluate these nuances through human-curated counterfactual pairs and multi-sentence segments. Our evaluation of state-of-the-art models reveals a persistent "quality gap" where translations of feminine subjects are less accurate than masculine ones, and commercial systems often fail to utilize inter-sentential context, achieving only 50-60% accuracy on contextual disambiguation.

Introduction

While Neural Machine Translation (NMT) has achieved high levels of fluency, it remains prone to errors regarding fine-grained linguistic features, particularly gender. Most modern NMT systems are trained on massive, uncurated web crawls that reflect and amplify societal biases. This leads to models that rely on stereotypical associations—such as assuming a "doctor" is masculine—or models that ignore surrounding context entirely when determining the correct gendered forms in morphologically rich languages.

Existing benchmarks, such as WinoMT or MuST-SHE, have laid the groundwork for studying these biases but often rely on synthetic templates or narrow professional contexts. MT-GenEval addresses these limitations by providing a dataset derived from natural language that requires both counterfactual reasoning and contextual understanding across multiple sentences.

The MT-GenEval Dataset

MT-GenEval is a human-curated benchmark consisting of 4,008 natural segments translated from English into eight target languages: Arabic, French, German, Hindi, Italian, Portuguese, Russian, and Spanish. The dataset is built upon two primary pillars:

1. Counterfactual Generation

To ensure a perfectly balanced evaluation, researchers extracted natural sentences from Wikipedia describing individuals. Human annotators then created "counterfactual" versions by swapping all gender-referring terms (e.g., changing "he" to "she" and adjusting names where necessary). This approach ensures that any variance in model performance is strictly attributable to gender rather than the complexity or content of the sentence.

2. Contextual Disambiguation

Unlike single-sentence benchmarks, MT-GenEval includes segments where the gender of a subject in the target sentence can only be determined by referring to the preceding sentences. This specifically tests a model’s ability to handle inter-sentential agreement, a known weakness in standard NMT architectures that often process sentences in isolation.

Evaluation Methodology

The authors propose an Automatic Accuracy Metric to facilitate rapid iteration. By comparing model outputs against both "correct" and "contrastive" (incorrect gender) references, the metric identifies if a model uses words unique to the incorrect gender. This automated approach has demonstrated a high correlation with human judgment, achieving F-scores of 0.80 or higher.

Key Insights and Results

The Gender Quality Gap

A significant finding of the study is that gender bias impacts more than just translation accuracy; it impacts overall translation quality. Models consistently produced lower BLEU scores for feminine-referring inputs compared to their masculine counterfactual counterparts, even when the underlying sentence structure was identical. This suggests that NMT models are fundamentally less robust when translating feminine contexts.

Performance of Commercial Systems

When evaluating three leading commercial MT systems, the researchers found that their ability to use context was surprisingly limited. On the contextual subset, these systems achieved accuracy rates between 50.5% and 61.3%—only slightly better than a random guess. This indicates that even state-of-the-art industrial models often rely on local patterns or stereotypes rather than cross-sentence logic.

Improvements via Gender-Filtered Training

The study demonstrates that these deficiencies can be mitigated. A specialized model trained with contextual data and gender-filtered self-training (GFST-CTX) significantly outperformed baselines. For example, on German contextual tasks, the GFST-CTX model reached 77.1% accuracy, compared to the 66.7% achieved by the baseline model.

Limitations and Future Directions

While MT-GenEval provides a more rigorous framework for gender evaluation, it currently focuses on binary gender (male/female). The authors acknowledge the necessity of expanding the dataset to include non-binary genders in future iterations. Additionally, as the dataset is currently English-centric, further research is required to understand how morphological gender in various source languages influences translation accuracy in the target language.

Conclusion

MT-GenEval highlights the ongoing challenges NMT systems face in achieving gender parity. By moving beyond synthetic templates to natural, contextual, and counterfactual data, this benchmark provides a more realistic measure of model performance. The results underscore the need for explicit gender-aware training strategies to close the quality gap and improve the reliability of machine translation in real-world scenarios.

Summary updated

BibTeX
@inproceedings{currey2022mt,
  author = {Anna Currey and Maria Nadejde and Raghavendra Reddy Pappagari and Mia Mayer and Stanislas Lauly and Xing Niu and Benjamin Hsu and Georgiana Dinu},
  title = {{MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=YDga7St-jK},
}
Figure