Abstract
Neural Machine Translation (NMT) systems built on top of Transformer-based architectures are routinely improving the state-of-the-art in translation quality according to word-overlap metrics. However, a growing number of studies also highlight the inherent gender bias that these models incorporate during training, which reflects poorly in their translations. In this work, we investigate whether these models can be instructed to fix their bias during inference using targeted, guided instructions as contexts. By translating relevant contextual sentences during inference along with the input, we observe large improvements in reducing the gender bias in translations, across three popular test suites (WinoMT, BUG, SimpleGen). We further propose a novel metric to assess several large pre-trained models (OPUS-MT, M2M-100) on their sensitivity towards using contexts during translation to correct their biases. Our approach requires no fine-tuning and thus can be used easily in production systems to de-bias translations from stereotypical gender-occupation bias. We hope our method, along with our metric, can be used to build better, bias-free translation systems.
Introduction
Modern Neural Machine Translation (NMT) models, while achieving high scores on traditional metrics like BLEU, often struggle with societal biases embedded in their training data. One of the most pervasive issues is stereotypical occupation bias, where models default to specific genders for certain professions—for instance, assuming a "doctor" is male and a "nurse" is female when translating into languages with grammatical gender like German or French.
Traditional mitigation strategies often involve fine-tuning models on balanced datasets. However, this approach is computationally expensive and relies on the availability of high-quality, gender-balanced parallel corpora, which are often scarce. This paper explores an alternative: can we leverage the "in-context learning" capabilities of NMT models to correct gender bias at inference time without any additional training?
Methodology
The researchers propose a de-biasing framework that operates entirely during the inference phase, utilizing targeted contextual instructions to guide the model's output.
1. Template-Based Context Injection
The core of the approach involves a "Template Bank" consisting of 87 unique English templates designed to provide unambiguous gender signals. Examples include phrases like "The doctor in the next sentence is a woman" or "There is a female librarian in the next phrase." These templates are prepended or appended to the source sentence.
A critical finding in the implementation was the choice of delimiter. While standard punctuation (periods or colons) often degraded translation quality, the researchers found that using a hash symbol (#) as a delimiter effectively separated the context from the source sentence while preserving the model's fluency and BLEU scores.
2. Greedy Search Strategy
To automate the de-biasing process, the authors implemented a "greedy" search strategy. For a given input, the system iteratively selects templates from the bank and generates translations. A morphological tagger (Stanza) is then used to verify if the output's grammatical gender matches the intended gender specified in the template. The process terminates as soon as a match is found, ensuring the most efficient path to a de-biased translation.
3. Context-Sensitivity Score (CSS)
To evaluate how effectively different models respond to these interventions, the authors introduced the Context-Sensitivity Score (CSS). This metric quantifies a model's "malleability"—its tendency to change its output based on the provided context—providing a standardized way to compare the responsiveness of various architectures to in-context guidance.
Experimental Setup
The framework was evaluated on three major gender-bias benchmarks:
- WinoMT: Focused on coreference resolution and occupation-based bias.
- BUG (Balanced-Bug): A large-scale corpus of real-world English sentences.
- SimpleGen: A dataset of gendered noun phrases with unambiguous ground truths.
The primary models tested were OPUS-MT and M2M-100 (a multilingual many-to-many model), with translations performed from English into German, French, and Spanish. General translation quality was monitored using the Tatoeba dataset.
Key Findings
Significant Bias Reduction
The results demonstrate that providing relevant context can dramatically improve translation accuracy. On the BUG dataset, the M2M-100 model's accuracy for German translations rose from a baseline of 58.13% to 87.36% when using the greedy strategy. These improvements were consistent across all tested target languages.
Model Sensitivity and Semantics
The study revealed that M2M-100 was generally more sensitive to context than OPUS-MT. Interestingly, models that exhibited higher initial bias were often more "malleable," making them better candidates for this de-biasing method. Furthermore, the models demonstrated a semantic understanding of the context; providing gender-irrelevant information (e.g., "The doctor works on weekends") did not reduce bias and often harmed translation quality.
Template Effectiveness
The structure of the templates played a significant role in performance. Longer templates with multiple gender signals (e.g., multiple pronouns) were more effective. Additionally, the "token distance"—the proximity of the gender signal to the occupation word—was a key factor; shorter distances generally led to better coreference resolution.
Conclusion
This research provides a practical, fine-tuning-free method for mitigating gender bias in NMT systems. By using relevant contextual templates and a greedy search strategy, models can be steered toward more equitable translations without sacrificing fluency. While currently limited to binary gender due to existing evaluation datasets, this work lays the foundation for more inclusive and controllable translation systems. Future work will focus on expanding to non-binary identities and automating the discovery of optimal templates through prompt mining.
Summary updated