On the Language Coverage Bias for Neural Machine Translation

Shuo Wang, Zhaopeng Tu, Zhixing Tan, Shuming Shi, Maosong Sun, Yang Liu

Abstract

Researchers have identified "Language Coverage Bias" (LCB) as a significant bottleneck in Neural Machine Translation (NMT). Unlike "translationese," which focuses on stylistic artifacts, LCB is content-dependent, arising from the fact that training data originating from the target language often lacks the thematic diversity of the source language. This paper proposes a framework to detect and mitigate LCB through bias-tagging and fine-tuning. Experimental results across multiple WMT20 tasks show that handling LCB leads to substantial gains in translation adequacy and BLEU scores.

Introduction: The Content Gap in Parallel Corpora

Most NMT models are trained on massive parallel corpora consisting of sentence pairs. However, these corpora are rarely homogeneous. They typically comprise two distinct types of data:

  1. Source-original data: Sentences originally written in the source language and translated into the target.
  2. Target-original data: Sentences originally written in the target language and translated back into the source.

While previous research has extensively studied "translationese"—the stylistic, content-independent quirks of translated text—this work focuses on Language Coverage Bias (LCB). LCB refers to the regional and cultural differences in what people discuss. For instance, English-original texts frequently mention "Trump" or "the US," whereas German-original texts are more likely to focus on local issues such as "refugees" or specific European cities.

Word clouds illustrating Language Coverage Bias: English-original text (left) focuses on US-centric topics, while German-original text (right) focuses on European and local issues.

When these data types are mixed indiscriminately, NMT models struggle to translate specific content words (nouns, verbs, adjectives) accurately, particularly when the test set is source-original. The authors found that target-original data often performs poorly at translating these specific content words, leading to a drop in overall translation adequacy.

Methodology: Detecting and Mitigating Bias

The researchers developed a three-step approach to address LCB:

  1. Detection: Since many large-scale datasets lack metadata regarding the original language, the authors trained language models on massive monolingual datasets to estimate the likelihood of a sentence being "original" to its language.
  2. Bias-Tagging: To prevent target-original data from "polluting" the primary translation task, the authors attached a special tag (e.g., <target-original>) to the source side of identified examples. This allows the model to learn the specific characteristics of biased data without conflating them with the target distribution.
  3. Fine-Tuning: Models were pre-trained on the full mixed dataset and then fine-tuned exclusively on high-quality source-original data to align the model's output more closely with the expected source-original distribution.

Key Insights and Results

One of the most striking findings was the "Less is More" revelation: training a model on only the source-original data (roughly 50% of the total dataset) yielded performance comparable to training on the full, mixed dataset. This suggests that target-original data, if not handled carefully, acts more like noise than useful signal for translating source-original text.

BLEU scores for En-Zh and Zh-En tasks comparing models trained on varying ratios of source-original (Most Src-Orig) vs. target-original (Most Tgt-Orig) data. Source-original data consistently leads to higher performance.

The proposed tagging and tuning methods consistently outperformed strong baselines, achieving an average improvement of +0.8 to +0.9 BLEU points across six translation directions. Furthermore, the methods significantly improved translation adequacy. In the En-Zh task, tagging improved the F-measure for noun translation from 70.7 to 72.3.

Conclusion and Future Work

Language Coverage Bias represents a fundamental challenge in NMT that goes beyond stylistic differences. By acknowledging the content-dependent nature of training data, researchers can build more robust models. Future work will explore how LCB manifests in closely related language families and investigate more sophisticated multi-task learning frameworks to handle the bias more effectively.

Summary updated

BibTeX
@inproceedings{wang2021language,
  author = {Shuo Wang and Zhaopeng Tu and Zhixing Tan and Shuming Shi and Maosong Sun and Yang Liu},
  title = {{On the Language Coverage Bias for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.422},
}
Figure