Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques

Joel Escudé Font, Marta R. Costa-jussàACL 2019

TL;DR

This paper tackles the issue of gender bias in Neural Machine Translation (NMT) by proposing a method to equalize these biases using debiased word embeddings. By integrating pre-trained, gender-neutral word embeddings into the Transformer architecture, the authors demonstrate improvements in translation quality and a reduction in gender stereotyping, particularly for occupations.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems, while achieving impressive performance, often inherit and even amplify societal biases present in their training data. One significant bias is gender bias, where translations can reflect stereotypical associations between genders and certain professions or roles. For instance, the English word "friend" might be translated into Spanish as "amigo" (male) or "amiga" (female) based on the perceived gender of the associated occupation, rather than the neutral meaning of "friend." This paper aims to address this by investigating how to mitigate gender bias in NMT by leveraging techniques developed for debiasing word embeddings.

Methods

The core of this work lies in adapting existing word embedding debiasing techniques for use within an NMT framework. The authors employ the Transformer architecture, a state-of-the-art NMT model. Their methodology involves:

  • Pre-trained Word Embeddings: They utilize pre-trained word embeddings, specifically GloVe, as a foundation.
  • Debiasing Techniques: They experiment with two primary debiasing approaches:
  • Hard-Debiased GloVe (Bolukbasi et al., 2016): A post-processing method that identifies and neutralizes gender-specific directions in the embedding space.
  • GN-GloVe (Zhao et al., 2018b): An algorithm that modifies the training process of GloVe to learn gender-neutral representations from the outset.
  • Integration into NMT: These debiased embeddings are then integrated into the Transformer model in three configurations: in the encoder only, in the decoder only, or in both. The pre-trained embeddings are updated during the NMT training process.

Datasets

The study uses standard datasets for training and evaluation, along with a custom-created test set:

  • Training Data: The WMT English-Spanish benchmark dataset, comprising approximately 16.6 million sentences.
  • Validation and Test Sets: The newstest2012 and newstest2013 datasets from the WMT workshop for standard translation quality evaluation.
  • Occupations Test Set: A newly developed test set containing sentences with occupation-related contexts designed to specifically evaluate gender bias. This set includes sentences with the pattern "I've known {her, him, <proper noun>} for a long time, my friend works as {a, an} <occupation>." The occupations are sourced from the U.S. Bureau of Labor Statistics.

Key Insights and Ideas

  • Word Embeddings as a Bottleneck: The paper highlights that biases in word embeddings, which are fundamental components of NMT systems, directly impact translation output and can lead to gender stereotyping.
  • Debiasing for NMT is Feasible: The study demonstrates that debiasing techniques applied to word embeddings can be effectively integrated into NMT models without significantly degrading overall translation quality (measured by BLEU).
  • Context is Key for Gender Neutrality: The custom "Occupations test" set reveals that when NMT systems are less gender-biased, they are better at disregarding stereotypical associations and correctly translating gender-neutral terms like "friend" based on context.
  • GN-GloVe Shows Promise: The GN-GloVe embeddings, which learn gender neutrality during training, appear to offer a slight advantage in reducing bias while maintaining or even improving translation quality in certain configurations.

Key Results

  • Translation Quality (BLEU): Using pre-trained word embeddings generally improves BLEU scores compared to the baseline system (which learns embeddings from scratch). Debiased embeddings, particularly GN-GloVe, maintain or slightly increase this improvement. For example, GN-GloVe in both encoder and decoder achieved a BLEU score of 30.74, outperforming the baseline of 29.78.
  • Gender Bias Reduction: The "Occupations test" set shows significant improvements in correctly translating the ambiguous word "friend." For instance, when the pronoun is "her," models using debiased embeddings show a higher percentage of correct "amiga" translations compared to the baseline. The paper notes that male pronouns are generally predicted with high accuracy across models, but female pronouns see more varied performance, with debiased models showing better results. For example, the GN-GloVe (Enc./Dec.) model achieved 56.4% accuracy in predicting "amiga" for the "her" pronoun in the context of "mine shuttle car operator," a notable improvement over some other models.

Limitations and Future Work

  • Limited Scope of Bias: The study focuses primarily on gender bias in the context of news articles and professions. The authors acknowledge that human language corpora contain a much broader spectrum of biases (e.g., related to race, religion, industrial, medical, or legal domains) that require further investigation.
  • Language Pair Specificity: The experiments are conducted on the English-Spanish language pair. Future work could explore other language pairs with varying grammatical gender specifications.
  • Further Exploration of Debiasing: The authors suggest exploring other debiasing techniques and their impact on NMT. They also propose investigating how to incorporate gender information more explicitly into NMT systems, potentially through data augmentation or architectural changes.
  • Broader Social Constructs: The paper points to the need to study other social constructs and stereotypes beyond gender, such as race and religious beliefs, in NMT.

Summary updated

BibTeX
@inproceedings{font2019equalizing,
  author = {Joel Escudé Font and Marta R. Costa-jussà},
  title = {{Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure