TL;DR
A decade after the initial call to address gender bias in machine translation (MT), this review finds that while research has grown, a simple technical fix remains elusive. The field is grappling with the rise of large language models (LLMs), which present new opportunities and challenges in achieving truly inclusive and context-aware translation.
What Problem Does This Paper Address?
The paper addresses the persistent problem of gender bias in machine translation (MT). This bias can lead to the misrepresentation of individuals, reinforce harmful stereotypes, and create tangible disparities in service for marginalized groups. Despite initial optimism for a quick technological solution, the authors argue that gender bias is a complex, multifaceted issue deeply intertwined with societal inequalities, and it has proven resistant to simple fixes. The paper aims to take stock of the last decade of research, identify trends and persistent gaps, and discuss the implications of emerging LLMs for this ongoing challenge.
Methods
The authors conducted a systematic review of over 100 papers on gender bias in machine translation, primarily drawing from the ACL Anthology. Their methodology involved:
- ACL Anthology Search: Querying the ACL Anthology for papers published between 2014 and 2024 using keywords like "translation," "gender," and "bias."
- Manual Selection and Annotation: Manually filtering the search results based on specific criteria (focus on MT, human entities, gender fairness) and annotating the selected papers across 11 fields to capture conceptualizations of bias, trends, and experimental design.
- Diachronic Analysis: Analyzing the trends and evolution of research approaches over time, including the focus on non-binary gender, types of mitigation strategies, and technological paradigms.
- Literature Synthesis: Synthesizing findings from the reviewed papers to identify key trends, persistent gaps, and emerging challenges, particularly in the context of LLMs.
Datasets
The paper does not introduce new datasets. Instead, it reviews existing research that utilizes various datasets for training and evaluation. Table 2 lists several popular MT gender bias benchmarks, including:
- Equity evaluation corpus
- WinoMT
- Occupation test set
- Arabic parallel gender corpus
- Must-SHE
- SimpleGen
- BUG
- MT-GenEval
- GENTE
- Multilingual holistic bias
- MiTTenS
These benchmarks vary in their focus (e.g., pronouns, occupations, stereotypes) and data type (artificial vs. natural).
Key Insights and Ideas
- Bias is Dynamic and Multifaceted: The initial optimism for a quick technical fix has given way to a more nuanced understanding that bias is dynamic, resistant to simple solutions, and deeply embedded in societal structures.
- LLMs Present New Frontiers: The rise of LLMs offers new opportunities for multilingual and multimodal translation but also introduces new challenges in addressing bias, particularly concerning context and user-centricity.
- English-Centricity Persists: Despite the goal of multilinguality, much research remains heavily focused on English and a few Western languages, risking the perpetuation of anglocentric biases.
- Beyond the Binary is Growing: There's an increasing trend towards research that acknowledges and attempts to represent non-binary gender identities, moving beyond a strictly binary treatment of gender.
- Lack of Intersectional Analysis: A significant gap exists in research that considers the intersection of gender with other identity categories (e.g., race, sexual orientation), which is crucial for understanding complex discrimination.
- Human Engagement is Scarce: A severe lack of human evaluation and user involvement in bias assessment is noted, hindering a true understanding of real-world harms and user needs.
- Sentence-Level Focus Limits Scope: Most research still focuses on sentence-level translation, which is insufficient for capturing gender nuances that often require broader context.
Key Results
- Growing Research Interest: The number of publications on gender bias in MT has steadily increased over the past decade, with a notable surge from 2019 to 2023.
- Dominance of Text-to-Text: The vast majority of studies (123 papers) focus on text-to-text translation, with limited research in speech or image-guided translation.
- Sentence-Level Dominance: 118 papers focus on sentence-level translation, with paragraph or document-level studies being exceptional.
- Mitigation Strategies are Common: Nearly half of the reviewed papers (51) propose a dedicated mitigation strategy, but no single approach has emerged as a clear winner.
- Limited Language Pair Coverage: Research is heavily concentrated on a few language pairs, primarily English-to-German, English-to-French, and English-to-Spanish.
- Binary Treatment of Gender: The majority of papers (105 out of 133) treat gender as a binary category.
- Limited Human Evaluation: Only 40 works rely on human evaluation to measure bias, with a significant portion of those being manual evaluations by expert linguists rather than broader user surveys.
Limitations and Future Work
Limitations:
- ACL Anthology Focus: The review is limited to peer-reviewed papers in the ACL Anthology, potentially missing relevant work from other sources.
- Author Positionality: The authors acknowledge that their European background and "White" identity may have influenced their perspective and selection of examples, potentially not fully capturing the experiences of underrepresented groups.
- Anglocentric Bias: The review itself, while aiming to address anglocentricity, is still influenced by the prevalence of English-centric research in the field.
Future Work:
- Multilinguality and Multiculturalism: Expand research to a wider range of languages and cultures, moving beyond anglocentric approaches and acknowledging diverse cultural contexts of gender.
- Beyond Textual Modality: Investigate bias in multimodal MT systems (speech, vision) and how it intersects with other identity axes.
- User-Centric and Realistic Assessment: Prioritize human engagement, user studies, and realistic, context-aware evaluation methods that go beyond artificial sentence-level tests.
- Handling Non-Binary and Neopronouns: Develop more robust methods for translating emerging non-binary linguistic expressions and neopronouns, leveraging LLMs' in-context learning capabilities.
- Intersectional Bias: Conduct more research on the intersection of gender with other identity categories to understand complex discrimination.
- Personalization with Ethical Considerations: Explore personalization of MT systems to cater to individual user preferences while carefully considering ethical and privacy implications.
- Data Curation and Model Behavior: Investigate the connection between training data, model behavior, and the representation of marginalized identities, acknowledging the challenges posed by large, undisclosed pre-training datasets.
Summary updated