Abstract
Gender bias in machine translation (MT) is often discussed through abstract model-centric metrics. This paper, presented at EMNLP 2024, shifts the focus to "allocative harm"—the tangible disadvantage experienced by users. Through a large-scale human-centered study involving 88 participants, we quantify the "gender tax" associated with correcting biased MT outputs. Our results show that translating into feminine forms takes twice as long and requires four times the technical effort compared to masculine forms. These findings highlight significant economic and workload disparities that current automatic evaluation metrics fail to capture.
Introduction
While it is well-documented that MT systems often default to masculine forms, the practical consequences for human users have remained largely unquantified. This study moves beyond accuracy scores to measure the actual time, effort, and financial cost incurred when humans must correct gender-biased translations. By focusing on the post-editing process, the researchers isolate gender as the primary variable affecting human workload.
Quantifying Allocative Harm
The authors argue that gender bias leads to a lower quality of service for specific groups. For instance, a professional woman translating her biography must exert more effort to correct masculine defaults than a man in the same position. This "extra work" constitutes a tangible harm that can be measured in terms of technical effort (edits) and temporal effort (time). This study aims to bridge the gap between model performance and human experience.
Experimental Methodology
The researchers conducted a controlled post-editing (PE) experiment with 88 participants, including professional translators and students. Using the Matecat tool, they tracked:
- Time to Edit (TE): The total time spent on each sentence.
- Human-targeted Translation Edit Rate (HTER): A measure of technical effort based on the number of insertions, deletions, and substitutions.
The study utilized three datasets (MT-GenEval, MuST-SHE, and MTGEN-UN) across English to Italian, Spanish, and German, focusing on sentences where gender cues were either ambiguous or explicitly provided in the source text.
Results: The "Gender Tax" in Post-Editing
The study reveals a massive disparity in the effort required to produce feminine versus masculine translations across all tested languages and datasets.
Key Findings:
- The Effort Gap: On average, feminine post-editing took twice as long and required four times more editing operations (HTER) than masculine tasks.
- The Student Penalty: The burden was particularly high for non-professional users. For Italian students, the effort for feminine translations spiked by +330% in time and +636% in HTER compared to masculine tasks.
- Metric Blindness: Standard automatic metrics like BLEU showed a very weak correlation (-0.18) with actual human time-to-edit, suggesting they are poor proxies for human cost.
- Economic Inequity: These disparities translate directly into financial costs. If translators are paid by the hour, feminine content is more expensive to produce; if paid per word, the translator's effective hourly rate drops significantly when working on feminine-themed text.
Discussion and Future Work
The findings underscore the need for evaluation frameworks that account for human effort rather than just string similarity. Current MT systems impose a substantial "gender tax" on users and translators dealing with feminine content.
While this study focused on binary gender in Western European languages, the authors acknowledge the importance of expanding this research to non-binary forms and lower-resource languages. They advocate for participatory research and the development of new metrics that better predict the actual human labor required to achieve fair and accurate translations.
Summary updated