Abstract
Neural Machine Translation (NMT) models are typically trained on parallel corpora under the assumption of semantic equivalence. However, real-world datasets often contain a mix of source-original and target-original data, leading to "language coverage bias." This content-driven mismatch occurs because different cultures discuss distinct topics, resulting in vocabulary and entity distributions that vary by the original language of the text. This paper introduces a framework to automatically detect these biases and proposes mitigation strategies—bias-tagging and fine-tuning—that significantly improve translation quality and content accuracy across major language pairs.
Introduction
In NMT research, much attention has been paid to "translationese," the stylistic simplification found in translated text. This study identifies a more fundamental issue: language coverage bias. Unlike stylistic bias, coverage bias is content-dependent. For instance, a German-original text is more likely to discuss local European politics than an English-original text. When models are trained indiscriminately on mixed data, they often struggle to preserve specific meaning (adequacy), leading to translations that are fluent but omit crucial details or named entities.
Methodology
The authors propose a two-step framework to address this bias:
1. Detection
Since most large-scale datasets lack metadata regarding the original language, the authors developed an automatic detection method. By training Language Models (LMs) on large monolingual datasets for both source and target languages, they can score the "naturalness" of each side of a parallel pair.
As shown in the table above, the LM-based approach achieves high F1 scores, particularly in the English-Japanese (En-Ja) pair, outperforming standard classification baselines.
2. Mitigation
Once data is categorized, two strategies are employed:
- Bias-Tagging: Prepending a special tag (e.g.,
<target-original>) to the source text during training to help the model distinguish between data types. - Fine-Tuning: Pre-training on the full corpus followed by fine-tuning exclusively on high-quality source-original data.
Experimental Setup
The study utilized WMT20 benchmarks for English-German (En-De), English-Chinese (En-Zh), and English-Japanese (En-Ja), totaling over 75 million sentence pairs. Evaluation was conducted on newstest2020 to ensure alignment with modern NMT standards.
Key Results and Insights
The "Less is More" Phenomenon
A critical finding of the research is that training on only the source-original portion of a dataset (often just 50% of the total) can yield performance comparable to or better than using the entire mixed dataset. This indicates that target-original data can act as noise, degrading the model's ability to handle complex content words.
Performance Gains
The proposed mitigation methods consistently outperformed strong Transformer-Big baselines. In the En $\rightarrow$ Zh direction, the authors observed improvements of up to 1.1 BLEU points. Furthermore, the models showed significantly higher accuracy in translating "content words" (nouns, verbs, and adjectives), which are typically the most affected by coverage bias.
Adequacy vs. Fluency
The research highlights that while target-original data might help a model sound more natural (fluency), it frequently causes the model to fail at capturing specific meanings (adequacy). This explains why techniques like back-translation, which inherently generate target-original data, can sometimes negatively impact performance on standard test sets.
Conclusion
Language coverage bias represents a significant hurdle for high-fidelity NMT. By explicitly modeling the distinction between source-original and target-original data, researchers can produce models that are not only fluent but also more accurate in preserving the rich informational content of the source text. Future work will explore these biases in low-resource settings and within massive multilingual models.
Summary updated