How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts

Shanya Sharma, Manan Dey, Koustuv Sinha

Introduction

Neural Machine Translation (NMT) models, while powerful, often reflect and amplify societal biases found in their training data. A prominent issue is stereotypical gender-occupation bias, where models default to masculine or feminine pronouns based on professional stereotypes (e.g., "doctor" as male, "nurse" as female) regardless of the source text's intent. This paper explores a novel, inference-time solution: "steering" models toward unbiased translations by providing relevant contextual sentences as guided instructions.

The De-biasing Pipeline

The researchers propose a pipeline that operates entirely during the inference phase, avoiding the high costs of model fine-tuning. The process involves four key stages:

  1. Bias Detection: A morphological tagger (Stanza) identifies the gender of entities in the source and the initial translation. A mismatch triggers the de-biasing process.
  2. Context Injection: The system selects a gender-explicit template from a bank of 87 hand-crafted sentences (e.g., "There is a female librarian in the next phrase...").
  3. Translation with Context: This template is prepended or appended to the input, separated by a specific delimiter (e.g., "#").
  4. Post-processing: After translation, the system removes the translated context, leaving the corrected target sentence.
Overview of the de-biasing pipeline showing context injection for a female librarian.

Experimental Setup

The method was evaluated across English to German (de), French (fr), and Spanish (es) using two NMT architectures: M2M-100 (a many-to-many model) and OPUS-MT (bilingual models). Performance was measured on three benchmarks:

  • WinoMT: Standard gender bias benchmark.
  • BUG (Balanced-Bug): Real-world English sentences.
  • SimpleGen: Artificially generated, unambiguous gendered phrases.

Model Sensitivity and Semantic Signals

A core contribution of the study is the introduction of the Context-Sensitivity Score (CSS). The researchers found that models vary significantly in how they respond to extra context. M2M-100 demonstrated higher sensitivity than OPUS-MT, making it more prone to initial bias but also more responsive to the de-biasing templates.

Comparison of context sensitivity across models (OPUS-MT and M2M-100), languages, and datasets.

Crucially, the models were not merely reacting to noise; "gender-irrelevant" contexts actually degraded performance, proving that the models leverage the specific semantic gender signals within the templates. The researchers also noted a bias asymmetry: models found it harder to correct female stereotypes (e.g., a male nurse) than male ones, likely due to male-centric training data. Additionally, the choice of delimiter proved vital: using a hash ("#") preserved translation quality (BLEU scores) better than standard punctuation.

Key Results

The approach yielded substantial improvements in gender translation accuracy. For instance, on the BUG dataset with M2M-100, accuracy rose from a baseline of approximately 58% to 87.36% using a greedy template selection strategy.

The researchers also demonstrated efficiency: even a small subset of 5 random templates provided significant bias reduction. This suggests the method is practical for real-time systems where testing all 87 templates would be computationally prohibitive.

Limitations and Future Work

While effective, the current pipeline is limited to binary gender identities. Future research is needed to address non-binary and transgender identities. Furthermore, the system's reliance on morphological taggers introduces a dependency on tagger accuracy. Future iterations could explore dynamic, automated prompt generation to replace the fixed template bank and reduce inference latency.

Summary updated

Abstract

Neural Machine Translation systems built on top of Transformer-based architectures are routinely improving the state-of-the-art in translation quality according to word-overlap metrics. However, a growing number of studies also highlight the inherent gender bias that these models incorporate during training, which reflects poorly in their translations. In this work, we investigate whether these models can be instructed to fix their bias during inference using targeted, guided instructions as contexts. By translating relevant contextual sentences during inference along with the input, we observe large improvements in reducing the gender bias in translations, across three popular test suites (WinoMT, BUG, SimpleGen). We further propose a novel metric to assess several large pre-trained models (OPUS-MT, M2M-100) on their sensitivity towards using contexts during translation to correct their biases. Our approach requires no fine-tuning and thus can be used easily in production systems to de-bias translations from stereotypical gender-occupation bias 1. We hope our method, along with our metric, can be used to build better, bias-free translation systems.
BibTeX
@inproceedings{sharma2022how,
  author = {Shanya Sharma and Manan Dey and Koustuv Sinha},
  title = {{How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=MjBtsDEwHh},
}
Figure