Abstract
Researchers have identified "Language Coverage Bias" (LCB) as a significant bottleneck in Neural Machine Translation (NMT). Unlike "translationese," which focuses on stylistic artifacts, LCB is content-dependent, arising from the fact that training data originating from the target language often lacks the thematic diversity of the source language. This paper proposes a framework to detect and mitigate LCB through bias-tagging and fine-tuning. Experimental results across multiple WMT20 tasks show that handling LCB leads to substantial gains in translation adequacy and BLEU scores.
Introduction: The Content Gap in Parallel Corpora
Most NMT models are trained on massive parallel corpora consisting of sentence pairs. However, these corpora are rarely homogeneous. They typically comprise two distinct types of data:
- Source-original data: Sentences originally written in the source language and translated into the target.
- Target-original data: Sentences originally written in the target language and translated back into the source.
While previous research has extensively studied "translationese"—the stylistic, content-independent quirks of translated text—this work focuses on Language Coverage Bias (LCB). LCB refers to the regional and cultural differences in what people discuss. For instance, English-original texts frequently mention "Trump" or "the US," whereas German-original texts are more likely to focus on local issues such as "refugees" or specific European cities.
When these data types are mixed indiscriminately, NMT models struggle to translate specific content words (nouns, verbs, adjectives) accurately, particularly when the test set is source-original. The authors found that target-original data often performs poorly at translating these specific content words, leading to a drop in overall translation adequacy.
Methodology: Detecting and Mitigating Bias
The researchers developed a three-step approach to address LCB:
- Detection: Since many large-scale datasets lack metadata regarding the original language, the authors trained language models on massive monolingual datasets to estimate the likelihood of a sentence being "original" to its language.
- Bias-Tagging: To prevent target-original data from "polluting" the primary translation task, the authors attached a special tag (e.g.,
<target-original>) to the source side of identified examples. This allows the model to learn the specific characteristics of biased data without conflating them with the target distribution. - Fine-Tuning: Models were pre-trained on the full mixed dataset and then fine-tuned exclusively on high-quality source-original data to align the model's output more closely with the expected source-original distribution.
Key Insights and Results
One of the most striking findings was the "Less is More" revelation: training a model on only the source-original data (roughly 50% of the total dataset) yielded performance comparable to training on the full, mixed dataset. This suggests that target-original data, if not handled carefully, acts more like noise than useful signal for translating source-original text.
The proposed tagging and tuning methods consistently outperformed strong baselines, achieving an average improvement of +0.8 to +0.9 BLEU points across six translation directions. Furthermore, the methods significantly improved translation adequacy. In the En-Zh task, tagging improved the F-measure for noun translation from 70.7 to 72.3.
Conclusion and Future Work
Language Coverage Bias represents a fundamental challenge in NMT that goes beyond stylistic differences. By acknowledging the content-dependent nature of training data, researchers can build more robust models. Future work will explore how LCB manifests in closely related language families and investigate more sophisticated multi-task learning frameworks to handle the bias more effectively.
Summary updated