Abstract
Neural Machine Translation (NMT) often suffers from "under-translation," where parts of the source sentence are missing in the output. This research identifies translation entropy—the degree of uncertainty in translating a specific source word—as a key factor behind this problem. We demonstrate that high-entropy words are frequently discarded during beam search decoding. To address this, we introduce a coarse-to-fine framework that utilizes pseudo-target tokens to reduce initial task complexity. Among the proposed strategies, a two-pass method incorporating a disambiguation network achieves significant improvements in BLEU scores and alignment accuracy across Chinese-English, English-Japanese, and English-German tasks.
The Entropy-Under-Translation Link
Despite the success of NMT, models often suffer from under-translation, where parts of the source sentence are simply ignored in the output. While previous research focused on architectural fixes like coverage models, this paper investigates the nature of the words themselves.
The authors discover that the difficulty of a word is tied to its translation entropy. If a source word can be translated into many different target words with similar probabilities, the NMT model struggles to align it correctly. During beam search decoding, these high-entropy candidates often have lower individual probabilities, making them prime candidates for being pruned or "lost" in favor of more certain neighbors.
As shown in the empirical analysis, the ratio of under-translation increases almost linearly with translation entropy across different architectures, including RNNs and Transformers.
Coarse-to-Fine Framework
To help models focus on these high-entropy words, the authors propose a coarse-to-fine framework consisting of two main phases:
1. Coarse-grained Phase: Entropy Reduction
First, translation entropy is calculated for every word in the source vocabulary using a statistical lexicon. Words exceeding an entropy threshold are labeled "high-entropy." The authors then create pseudo-target sentences where all possible valid translations of a high-entropy source word are replaced by a single, unique "pseudo-token." This effectively collapses the entropy of that word to zero, simplifying the initial translation task.
2. Fine-grained Phase: Model Integration
Three strategies were proposed to integrate this simplified data back into NMT training:
- Pre-training: Initializing the model on pseudo-data before fine-tuning on real data.
- Multitask Learning: Simultaneously predicting both real and pseudo-target sentences.
- Two-pass Method: The most effective approach, where the model first translates the source into a pseudo-target sentence, followed by a disambiguation network that predicts the specific target word using internal hidden states.
Experimental Results
The researchers evaluated their methods on three major translation tasks: Chinese-to-English (LDC corpus), English-to-Japanese (KFTT), and English-to-German (WMT 2014).
- BLEU Score Gains: The "Two-pass" method consistently outperformed baselines. On Chinese-to-English, it improved the baseline RNMT by 1.21 BLEU points and the state-of-the-art Transformer by 0.64 BLEU points.
- Reduced Under-translation: Manual analysis showed that for high-entropy words, the under-translation ratio dropped from 8.05% to 4.86% when using the two-pass method.
- Better Alignment: The models showed significantly lower Alignment Error Rates (AER), indicating that the internal attention mechanism became more accurate at identifying source words.
Conclusion and Future Work
By using pseudo-tokens, the authors teach the model "where" something important belongs before forcing it to decide "what" it is. This separation of location and identity significantly mitigates the dropout of complex words.
However, the method currently relies on external statistical lexicons to calculate entropy. Future work could explore end-to-end entropy estimation within the NMT model itself. Additionally, while the two-pass method is highly effective, it introduces extra computational complexity during decoding and requires careful hyperparameter tuning for the entropy threshold and multitask weights.
Summary updated