Addressing the Under-Translation Problem from the Entropy Perspective

Yang Zhao, Jiajun Zhang, Chengqing Zong, Zhongjun He, Hua Wu

Abstract

Neural Machine Translation (NMT) often suffers from "under-translation," where parts of the source sentence are missing in the output. This research identifies translation entropy—the degree of uncertainty in translating a specific source word—as a key factor behind this problem. We demonstrate that high-entropy words are frequently discarded during beam search decoding. To address this, we introduce a coarse-to-fine framework that utilizes pseudo-target tokens to reduce initial task complexity. Among the proposed strategies, a two-pass method incorporating a disambiguation network achieves significant improvements in BLEU scores and alignment accuracy across Chinese-English, English-Japanese, and English-German tasks.

Despite the success of NMT, models often suffer from under-translation, where parts of the source sentence are simply ignored in the output. While previous research focused on architectural fixes like coverage models, this paper investigates the nature of the words themselves.

The authors discover that the difficulty of a word is tied to its translation entropy. If a source word can be translated into many different target words with similar probabilities, the NMT model struggles to align it correctly. During beam search decoding, these high-entropy candidates often have lower individual probabilities, making them prime candidates for being pruned or "lost" in favor of more certain neighbors.

The relationship between translation entropy and the under-translation ratio across manual and automatic analyses.

As shown in the empirical analysis, the ratio of under-translation increases almost linearly with translation entropy across different architectures, including RNNs and Transformers.

Coarse-to-Fine Framework

To help models focus on these high-entropy words, the authors propose a coarse-to-fine framework consisting of two main phases:

1. Coarse-grained Phase: Entropy Reduction

First, translation entropy is calculated for every word in the source vocabulary using a statistical lexicon. Words exceeding an entropy threshold are labeled "high-entropy." The authors then create pseudo-target sentences where all possible valid translations of a high-entropy source word are replaced by a single, unique "pseudo-token." This effectively collapses the entropy of that word to zero, simplifying the initial translation task.

2. Fine-grained Phase: Model Integration

Three strategies were proposed to integrate this simplified data back into NMT training:

  • Pre-training: Initializing the model on pseudo-data before fine-tuning on real data.
  • Multitask Learning: Simultaneously predicting both real and pseudo-target sentences.
  • Two-pass Method: The most effective approach, where the model first translates the source into a pseudo-target sentence, followed by a disambiguation network that predicts the specific target word using internal hidden states.
Architectural diagrams for the proposed (a) Pre-training, (b) Multitask, and (c) Two-pass methods.

Experimental Results

The researchers evaluated their methods on three major translation tasks: Chinese-to-English (LDC corpus), English-to-Japanese (KFTT), and English-to-German (WMT 2014).

  • BLEU Score Gains: The "Two-pass" method consistently outperformed baselines. On Chinese-to-English, it improved the baseline RNMT by 1.21 BLEU points and the state-of-the-art Transformer by 0.64 BLEU points.
  • Reduced Under-translation: Manual analysis showed that for high-entropy words, the under-translation ratio dropped from 8.05% to 4.86% when using the two-pass method.
  • Better Alignment: The models showed significantly lower Alignment Error Rates (AER), indicating that the internal attention mechanism became more accurate at identifying source words.

Conclusion and Future Work

By using pseudo-tokens, the authors teach the model "where" something important belongs before forcing it to decide "what" it is. This separation of location and identity significantly mitigates the dropout of complex words.

However, the method currently relies on external statistical lexicons to calculate entropy. Future work could explore end-to-end entropy estimation within the NMT model itself. Additionally, while the two-pass method is highly effective, it introduces extra computational complexity during decoding and requires careful hyperparameter tuning for the entropy threshold and multitask weights.

Summary updated

Abstract

Neural Machine Translation (NMT) has drawn much attention due to its promising translation performance in recent years. However, the under-translation problem still remains a big challenge. In this paper, we focus on the under-translation problem and attempt to find out what kinds of source words are more likely to be ignored. Through analysis, we observe that a source word with a large translation entropy is more inclined to be dropped. To address this problem, we propose a coarse-to-fine framework. In coarse-grained phase, we introduce a simple strategy to reduce the entropy of highentropy words through constructing the pseudo target sentences. In fine-grained phase, we propose three methods, including pre-training method, multitask method and two-pass method, to encourage the neural model to correctly translate these high-entropy words. Experimental results on various translation tasks show that our method can significantly improve the translation quality and substantially reduce the under-translation cases of high-entropy words.
BibTeX
@inproceedings{zhao2019addressing,
  author = {Yang Zhao and Jiajun Zhang and Chengqing Zong and Zhongjun He and Hua Wu},
  title = {{Addressing the Under-Translation Problem from the Entropy Perspective}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.3301451},
}
Figure