What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study

Beatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof Arenas, Luisa Bentivogli

Abstract

Gender bias in machine translation (MT) is often discussed through abstract model-centric metrics. This paper, presented at EMNLP 2024, shifts the focus to "allocative harm"—the tangible disadvantage experienced by users. Through a large-scale human-centered study involving 88 participants, we quantify the "gender tax" associated with correcting biased MT outputs. Our results show that translating into feminine forms takes twice as long and requires four times the technical effort compared to masculine forms. These findings highlight significant economic and workload disparities that current automatic evaluation metrics fail to capture.

Introduction

While it is well-documented that MT systems often default to masculine forms, the practical consequences for human users have remained largely unquantified. This study moves beyond accuracy scores to measure the actual time, effort, and financial cost incurred when humans must correct gender-biased translations. By focusing on the post-editing process, the researchers isolate gender as the primary variable affecting human workload.

Overview of the human cost associated with post-editing gender-biased MT outputs.

Quantifying Allocative Harm

The authors argue that gender bias leads to a lower quality of service for specific groups. For instance, a professional woman translating her biography must exert more effort to correct masculine defaults than a man in the same position. This "extra work" constitutes a tangible harm that can be measured in terms of technical effort (edits) and temporal effort (time). This study aims to bridge the gap between model performance and human experience.

Experimental Methodology

The researchers conducted a controlled post-editing (PE) experiment with 88 participants, including professional translators and students. Using the Matecat tool, they tracked:

  • Time to Edit (TE): The total time spent on each sentence.
  • Human-targeted Translation Edit Rate (HTER): A measure of technical effort based on the number of insertions, deletions, and substitutions.

The study utilized three datasets (MT-GenEval, MuST-SHE, and MTGEN-UN) across English to Italian, Spanish, and German, focusing on sentences where gender cues were either ambiguous or explicitly provided in the source text.

Results: The "Gender Tax" in Post-Editing

The study reveals a massive disparity in the effort required to produce feminine versus masculine translations across all tested languages and datasets.

Distribution of HTER (technical effort) and seconds per word (temporal effort) across datasets, languages, and user types.

Key Findings:

  1. The Effort Gap: On average, feminine post-editing took twice as long and required four times more editing operations (HTER) than masculine tasks.
  2. The Student Penalty: The burden was particularly high for non-professional users. For Italian students, the effort for feminine translations spiked by +330% in time and +636% in HTER compared to masculine tasks.
  3. Metric Blindness: Standard automatic metrics like BLEU showed a very weak correlation (-0.18) with actual human time-to-edit, suggesting they are poor proxies for human cost.
  4. Economic Inequity: These disparities translate directly into financial costs. If translators are paid by the hour, feminine content is more expensive to produce; if paid per word, the translator's effective hourly rate drops significantly when working on feminine-themed text.

Discussion and Future Work

The findings underscore the need for evaluation frameworks that account for human effort rather than just string similarity. Current MT systems impose a substantial "gender tax" on users and translators dealing with feminine content.

While this study focused on binary gender in Western European languages, the authors acknowledge the importance of expanding this research to non-binary forms and lower-resource languages. They advocate for participatory research and the development of new metrics that better predict the actual human labor required to achieve fair and accurate translations.

Summary updated

Abstract

Gender bias in machine translation (MT) is recognized as an issue that can harm people and society. And yet, advancements in the field rarely involve people, the final MT users, or inform how they might be impacted by biased technologies. Current evaluations are often restricted to automatic methods, which offer an opaque estimate of what the downstream impact of gender disparities might be. We conduct an extensive human-centered study to examine if and to what extent bias in MT brings harms with tangible costs, such as quality of service gaps across women and men. To this aim, we collect behavioral data from ~90 participants, who post-edited MT outputs to ensure correct gender translation. Across multiple datasets, languages, and types of users, our study shows that feminine post-editing demands significantly more technical and temporal effort, also corresponding to higher financial costs. Existing bias measurements, however, fail to reflect the found disparities. Our findings advocate for human-centered approaches that can inform the societal impact of bias.
BibTeX
@inproceedings{savoldi2024what,
  author = {Beatrice Savoldi and Sara Papi and Matteo Negri and Ana Guerberof Arenas and Luisa Bentivogli},
  title = {{What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1002},
}
Figure