gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena

Eva Vanmassenhove, Johanna MontiACL 2021

TL;DR

This paper introduces gENder-IT, a novel English-Italian challenge set designed to tackle the complexities of natural gender phenomena in machine translation. By providing word-level gender annotations and multiple gender-alternative translations, gENder-IT aims to improve MT systems' ability to handle cross-linguistic gender differences, a persistent challenge in achieving accurate and unbiased translations.

What Problem Does This Paper Address?

Machine translation (MT) systems often struggle with grammatical gender, a feature that varies significantly across languages. While some languages have grammatical gender for all nouns (e.g., French, Italian), others do not (e.g., English). This discrepancy means that information present in one language might be implicit or absent in another, leading to translation ambiguities. Specifically, when nouns refer to animate beings, their gender can correspond to biological sex (natural gender). However, MT systems, with their limited contextual understanding, often default to a single, potentially biased, translation for ambiguous cases. This paper addresses the lack of resources and challenge sets specifically designed to evaluate and improve MT's handling of these natural gender phenomena, particularly the disambiguation and correct translation of gendered referents across English and Italian.

Methods

The core contribution of this work is the creation of the gENder-IT challenge set. This set is derived from a subset of the MuST-SHE corpus, which consists of naturally occurring sentences from TED Talks. The authors performed a multi-stage process:

  • Corpus Cleaning: Sentences were processed to ensure consistency, removing missing parts, correcting spelling, and standardizing punctuation. This resulted in 694 annotated sentences.
  • Word-Level Gender Tagging: For English sentences, all (pro)nouns referring to people were annotated with gender tags. These tags include:
  • <F>: Female referent.
  • <M>: Male referent.
  • <A>: Ambiguous referent, where the gender cannot be determined from the context. Multiple ambiguous referents in a single sentence are further distinguished (e.g., <A1>, <A2>).
  • Exceptions were made for nouns that are inherently gender-specific (e.g., "mother") or for proper names, which were generally not considered gender clues unless the person was a public figure with readily available pronoun information.
  • Multiple Italian Translations: For sentences containing gender-ambiguous referents, multiple correct gender-alternative translations were provided for the Italian target side. This is crucial because Italian, being a pro-drop language, often omits subject pronouns, and gender agreement can appear in adjectives or past participles.

Datasets

  • MuST-SHE corpus: The gENder-IT challenge set is based on a subset of this corpus, specifically focusing on sentences from TED Talks.
  • gENder-IT: This is the newly introduced challenge set. It consists of 694 English-Italian parallel sentences, annotated with word-level gender tags on the English side and providing multiple gender-alternative translations for Italian where ambiguity exists. This set is distinct from previous work by focusing on natural sentences (not synthetic), covering a broader range of natural gender phenomena beyond just first-person singular agreement, and annotating multiple ambiguous referents within a single sentence.

Key Insights and Ideas

  • The "Generic Male" is a Problem: The paper highlights how the "generic male" form in languages like Italian, where the masculine is often used to refer to mixed-gender groups or professions, can obscure gender and lead to biased translations.
  • Italian's Pro-Drop Nature Adds Complexity: The fact that Italian often omits subject pronouns means that gender information might not be explicitly present in the translated sentence, making it harder to control for gender bias without careful annotation.
  • Ambiguity is Multifaceted: Gender ambiguity isn't just about pronouns; it can arise from nounless adjectives, nounless phrases, and bigender nouns (nouns with a single form for both genders). The gENder-IT set captures these nuances.
  • Proper Names Aren't Always Reliable: While names can sometimes hint at gender, they are not a definitive clue, especially across different cultures or when individuals' pronoun preferences differ from traditional associations.
  • Controlled Diversity is Key for MT Research: The authors emphasize the need for challenge sets that allow for controlled experimentation with specific linguistic features like gender, enabling researchers to develop more robust and less biased MT systems.

Key Results

  • The gENder-IT dataset contains 694 annotated English-Italian sentences.
  • Of these, 148 sentences required multiple gender-alternative translations in Italian.
  • The annotations include 138 <F> tags, 190 <M> tags, and 622 <A> tags (including disambiguated <A1>, <A2>, etc.).
  • The paper provides a detailed analysis of specific challenges in translating gender between English and Italian, including the handling of pro-drop subjects, past participles, bigender nouns, and the "generic masculine" form.

Limitations and Future Work

  • Limited Language Pair: The current gENder-IT set is specifically for English-Italian. The authors envision extending the corpus to other languages.
  • Identification of Ambiguity: While the dataset provides translations for ambiguous sentences, future work could focus on improving the automatic identification of gender-ambiguous sentences in English.
  • Paraphrasing for Neutrality: The authors suggest exploring paraphrasing techniques to generate more gender-neutral solutions, especially for sentences with multiple ambiguous referents.
  • Broader Phenomena: Future work could also involve expanding the scope to include other cross-linguistic gender phenomena beyond natural gender.

Summary updated

BibTeX
@inproceedings{vanmassenhove2021gender,
  author = {Eva Vanmassenhove and Johanna Monti},
  title = {{gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure