What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

Beatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof, Luisa Bentivogli

Introduction

For years, the discourse surrounding gender bias in Machine Translation (MT) has focused on "representational harms"—the ways in which AI systems reinforce stereotypes by defaulting to masculine forms for doctors and feminine forms for nurses. While these biases are well-documented through model-centric metrics like BLEU or COMET, these automatic scores act as opaque proxies. They tell us a model is biased, but they fail to quantify the actual friction, labor, and cost experienced by the human beings who interact with these systems.

A recent study by researchers at Fondazione Bruno Kessler and the University of Groningen, titled "What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study," shifts the focus toward "allocative harms." By measuring the time and effort required to correct biased outputs, the authors quantify a literal "gender tax" imposed on users of translation technology.

Methodology: Measuring the Human-in-the-Loop

To move beyond abstract statistics, the researchers conducted an extensive human-centered study involving 88 participants, ranging from professional translators to high school students. The core of the experiment was a post-editing (PE) task where participants were presented with English source sentences and their corresponding translations into Italian, Spanish, or German, generated by Google Translate.

The study employed a within-subjects design to isolate gender as the primary variable. Each participant edited the same sentences for both masculine and feminine target forms. The researchers tracked two behavioral metrics to determine the "cost" of bias:

  1. Time to Edit (TE): The total duration (in seconds) spent correcting a sentence.
  2. Human-targeted Translation Edit Rate (HTER): A measure of technical effort based on the number of insertions, deletions, and substitutions required to reach a correct translation.

Datasets

The study utilized two established benchmarks for gender translation, manually refined to ensure every sentence required a gendered choice:

  • MT-GenEval: Based on Wikipedia biographies, focusing on gendered occupational nouns.
  • MuST-SHE: Derived from TED talks, featuring first-person references and diverse gendered parts of speech, such as adjectives and past participles.

Key Findings: The Cost of the Masculine Default

The study’s results reveal a stark disparity in the service quality provided to different genders. Because state-of-the-art systems like Google Translate default to masculine forms, masculine translations often require zero edits. This creates a "free ride" for masculine content, while feminine content is relegated to a labor-intensive correction process.

The Effort Gap

The quantitative data highlights a significant imbalance in workload:

  • Temporal Effort: Across all tested languages and datasets, post-editing feminine translations took an average of 116% more time than masculine ones.
  • Technical Effort: The disparity in HTER was even more pronounced. Feminine translations required an average of 220% more editing operations. In specific instances, such as Italian translations in the MT-GenEval dataset, the effort required for feminine versions was four times higher than for masculine versions.

The Metric Disconnect

A critical insight from the study is that automatic metrics do not correlate well with human effort. A minor dip in an automatic score might mask a massive spike in human frustration and time spent. This suggests that current industry standards for measuring bias are fundamentally disconnected from the actual user experience.

The Economic "Gender Tax"

The researchers translated these effort metrics into economic terms. Under industry-standard "HTER-rate" pricing (where payment is based on effort), feminine translations are significantly more expensive to produce. Conversely, under "Word-rate" pricing (a flat fee per word), the financial burden shifts to the translator, who effectively earns a lower hourly wage when working on feminine-targeted content.

Discussion and Implications

The study also noted an "expertise buffer." While professional translators were faster at correcting bias than students, the relative gap between masculine and feminine effort remained consistent. Among lay users (the students), the bias often acted as a distraction; participants became so focused on fixing gender errors that they occasionally overlooked other translation inaccuracies.

Limitations

The authors acknowledge that this study focused on binary gender (masculine/feminine) due to the current lack of standardization for non-binary forms in the target languages. Additionally, the study focused on high-resource European languages and a single state-of-the-art MT system. Future research is needed to determine if these findings hold true for Large Language Models (LLMs) or lower-resource languages.

Conclusion

By putting a stopwatch and a price tag on gender bias, this research provides a powerful argument for a human-centered approach to AI ethics. Fairness in machine translation is not merely a social ideal; it is a prerequisite for equitable access to technology. As long as the "gender tax" exists, the benefits of AI will remain unevenly distributed, placing a disproportionate burden of labor and cost on those the system fails to represent by default.

Summary updated

BibTeX
@inproceedings{savoldi2024what,
  author = {Beatrice Savoldi and Sara Papi and Matteo Negri and Ana Guerberof and Luisa Bentivogli},
  title = {{What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1002},
}
Figure