A Taxonomy of Bias-Causing Ambiguities in Machine Translation

Michal MěchuraNAACL 2022

TL;DR

This paper introduces a novel taxonomy to systematically describe and categorize ambiguities in source texts that lead to biased machine translations, particularly concerning gender, number, and formality. By formalizing these ambiguities, the authors lay the groundwork for tools like Fairslator, which can identify these issues and prompt human users for disambiguation, ultimately leading to more accurate and less biased translations.

What Problem Does This Paper Address?

Machine translation (MT) systems, while increasingly sophisticated, often struggle with ambiguities present in source texts. When a source text leaves certain properties unspecified (like the gender of a person or the formality of address), but the target language requires these properties to be explicit, MT systems can make "unjustified assumptions." These assumptions, often based on patterns learned from training data, can lead to biased translations. For instance, translating "I am a doctor" into a language with gendered terms for "doctor" might default to the male form if the system has seen more male doctors in its training data, even though the source text provides no such information. This paper aims to address this by providing a structured way to identify and describe the root causes of such biases in MT.

Methods

The core contribution of this paper is the development of a taxonomy for describing unresolvable ambiguities in machine translation. This taxonomy operates by analyzing text pairs (source and translation) along three "axes":

  • Speaker Axis: Examines how the person speaking is represented, particularly if gender is introduced in the translation where it was absent in the source.
  • Listener Axis: Focuses on how the person being addressed is represented, looking for introduced gender, number, or formality.
  • Bystander Axis: Analyzes how third parties mentioned in the text are represented, specifically for introduced gender.

For each axis, the taxonomy defines:

  1. The possible readings allowed by the source text (e.g., "male or female").
  2. The reading actually expressed in the translation (e.g., "female").

This is achieved using a system of abbreviations (e.g., sm for singular masculine, sf for singular feminine, st for singular informal, sv for singular formal) to concisely represent these ambiguities and their resolutions. The paper also defines "unresolvable ambiguity" and "unjustified assumption" as technical concepts central to understanding MT bias.

Datasets

Not discussed in the paper. The paper focuses on the theoretical framework and the design of the taxonomy, rather than empirical evaluation on specific datasets. The tool Fairslator, which uses this taxonomy, is mentioned as a web-based tool for detecting and correcting bias.

Key Insights and Ideas

  • Bias as a Technical Problem: The paper frames bias in MT not just as a societal issue but as a technical consequence of unresolvable ambiguities and the unjustified assumptions MT systems make to resolve them.
  • The Power of Ambiguity: Unresolvable ambiguities are identified as the direct cause of biased translations. The taxonomy provides a systematic way to pinpoint these specific points of failure.
  • Three-Axis Framework: The speaker, listener, and bystander axes offer a comprehensive way to categorize how gender, number, and formality can be introduced or misrepresented during translation.
  • Human-in-the-Loop Necessity: The paper implicitly argues for the necessity of human intervention in disambiguation, as machines currently lack the contextual understanding to reliably resolve these ambiguities without bias. Tools like Fairslator are proposed as a bridge for this.
  • Universality of the Problem: The authors suggest that their definition of bias and the underlying technical issues apply not only to machines but also to human translators.

Key Results

The paper's primary "result" is the formal taxonomy itself, presented as a structured method for describing bias-causing ambiguities. It provides a clear framework and a set of descriptors that can be used to analyze translation pairs. The paper illustrates the application of this taxonomy with examples, showing how it can concisely represent the difference between the source text's ambiguity and the translation's resolved (and potentially biased) output. For instance, it demonstrates how "I am a doctor" (ambiguous) can lead to a translation where the speaker is identified as "sf" (singular feminine) in Czech, as shown in the example with "chtěla bych."

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future work:

  • I versus We: The current taxonomy assumes a single speaker axis that doesn't change grammatical number. It cannot handle texts where a speaker refers to themselves ("I") and a group they belong to ("we") within the same text.
  • Multiple Voices: The taxonomy assumes axes remain unchanged throughout a text. It struggles with dialogues or texts with multiple speakers where the "who is who" can change, leading to potential misattributions.
  • Word-Sense Ambiguities: The taxonomy is currently focused on gender, number, and formality. The authors suggest that ideally, it should cover all types of semantic properties affected by unresolvable ambiguities, including word-sense ambiguities (e.g., "river" having different meanings in French).
  • Gender-Neutral Language: The paper proposes expanding the taxonomy to include a "gender-neutral" value alongside male and female, acknowledging the existence and potential use of gender-neutral neologisms.
  • Broader Applicability: While the paper focuses on English-to-European language pairs, the underlying principles could be extended to other language combinations.

Summary updated

BibTeX
@inproceedings{mechura2022taxonomy,
  author = {Michal Měchura},
  title = {{A Taxonomy of Bias-Causing Ambiguities in Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure