Unmasking Biases: Exploring Gender Bias in English-Catalan Machine Translation through Tokenization Analysis and Novel Dataset

Audrey Mash, Carlos Escolano, Aleix Sant, Maite Melero, Francesca de Luca FornaciariCOLING 2024

TL;DR

This paper investigates how different ways of breaking down text (tokenization) affect gender bias in English-to-Catalan machine translation. They introduce a new dataset and find that while all methods show bias, some tokenizers are better at handling gendered language, especially when context is provided.

What Problem Does This Paper Address?

Machine translation (MT) systems, despite their impressive capabilities, often inherit and even amplify societal biases present in their training data. Gender bias is a particularly pervasive issue, leading to the over-representation of masculine forms and the under-representation or misrepresentation of feminine forms. This can perpetuate harmful stereotypes and create barriers for certain user groups. While much research focuses on debiasing the data itself, this paper explores a less-discussed but crucial factor: how the fundamental process of tokenization—breaking text into smaller units for the model to process—influences the manifestation of gender bias in translation. Specifically, they aim to understand if different tokenization strategies can mitigate or exacerbate gender bias in English-Catalan MT.

Methods

The core of this research lies in training and evaluating four different machine translation systems, each employing a distinct tokenization method:

  • Character-Based: Treats each character as a token.
  • Byte Pair Encoding (BPE): Iteratively merges frequent character pairs to create sub-word units.
  • SentencePiece Unigram: A subword regularization technique that learns token probabilities from data.
  • Morphological (Morfessor): Aims to preserve morphological information by segmenting words into meaningful units.

These systems were trained from scratch on a large parallel English-Catalan corpus. The researchers then evaluated both overall translation quality (using BLEU and COMET scores) and, crucially, gender accuracy. They also created a novel English-Catalan dataset specifically designed to test gendered translations.

Datasets

  • Training Data: A large parallel English-Catalan corpus derived from the Projecte Aina ca-en MT model, initially comprising 11.5 million sentence pairs, refined to 8.2 million pairs after filtering, and further augmented to 16.4 million pairs.
  • Evaluation Data: A synthetic English-Catalan version of the MuST-SHE dataset. This dataset was created by automatically translating the Spanish references of the original English-Spanish MuST-SHE corpus into Catalan. It contains segments where gender-neutral English words require gendered translations in Catalan, allowing for targeted evaluation of gender accuracy.

Key Insights and Ideas

  • Tokenization Matters: The way text is broken down significantly impacts gender bias in MT. This highlights that debiasing efforts need to consider not just the data but also the model's internal processing mechanisms.
  • Context is King (for Gender): When context is available, tokenization methods like BPE and SentencePiece Unigram perform better at correctly translating gendered terms. This suggests that models can leverage contextual cues more effectively with certain tokenization strategies.
  • Masculine as Default: Across all models and tokenization methods, there's a strong tendency to default to masculine forms when gender is ambiguous or not explicitly stated. This reflects biases present in the training data.
  • Stereotypes Amplified: The study suggests that existing stereotypes in the data are not just replicated but can be amplified in the translation output.

Key Results

  • Overall Translation Quality: SentencePiece Unigram achieved the highest BLEU scores, followed closely by BPE and Morfessor. Character-based tokenization performed the worst.
  • Gender Accuracy (with Context): BPE showed the highest gender translation accuracy (88.39%), followed by SentencePiece Unigram (82.13%). All models performed very well (over 98%) on translating masculine terms but struggled more with feminine terms.
  • Gender Accuracy (without Context): When context was absent, Morfessor generated the highest percentage of feminine forms (15.04%), while BPE had the lowest (10.35%). However, all models still produced a high percentage of masculine forms.
  • Stereotypical Translations: The study observed consistent translations of "nurse" to feminine forms and "doctor" to masculine forms, indicating the influence of societal stereotypes.

Limitations and Future Work

  • Limited Data for Morfessor: While Morfessor showed interesting results in context-free scenarios, the limited number of feminine gender terms it produced made detailed pattern identification difficult.
  • Holistic Strategies Needed: The authors emphasize that tokenization is just one piece of the puzzle. Future work should focus on holistic strategies that combine data-driven bias mitigation, architectural design choices, and tokenization methods.
  • Defining "Neutral" in Grammatical Gender Languages: The research highlights the ongoing debate about what constitutes a "neutral" form in languages like Catalan, which have grammatical gender, and how NLP can balance linguistic traditions with inclusivity.
  • New Dataset: The creation of the English-Catalan MuST-SHE dataset is a significant contribution, providing a valuable resource for future research on gender bias in Catalan MT.

Summary updated

BibTeX
@inproceedings{mash2024unmasking,
  author = {Audrey Mash and Carlos Escolano and Aleix Sant and Maite Melero and Francesca de Luca Fornaciari},
  title = {{Unmasking Biases: Exploring Gender Bias in English-Catalan Machine Translation through Tokenization Analysis and Novel Dataset}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure