Abstract
Standard Neural Machine Translation (NMT) models often "forget" which parts of a source sentence they have already translated, leading to repetitive or missing words. This paper introduces a coverage mechanism—a digital checklist that tracks translation history—to ensure every source word is accounted for, significantly improving both translation accuracy and alignment.
The Problem: Over-translation and Under-translation
When this research was conducted, NMT was rapidly overtaking traditional Statistical Machine Translation (SMT). However, it lacked a critical feature of SMT: a "coverage" mechanism. In older SMT systems, the decoder kept a strict record of which source words were "covered." In contrast, the then-state-of-the-art attention-based NMT models treated each step of translation as somewhat independent. The attention mechanism would "look" at the source sentence to decide what to translate next, but it didn't have a reliable memory of what it had already done.
This led to two major issues:
- Over-translation: The model gets stuck in a loop, translating the same source word multiple times (e.g., "close close").
- Under-translation: The model ignores certain words entirely, leaving parts of the original message out of the final translation.
Methods: Implementing Coverage and Fertility
The authors propose adding a coverage vector to the NMT architecture. This vector summarizes the attention history for each source word. They explore two primary ways to implement this:
- Linguistic Coverage Model: This is a "soft" version of the old SMT approach. It tracks the sum of past attention weights for each word and introduces Fertility—predicting how many target words a specific source word should produce based on its context.
- Neural Network (NN) Based Coverage Model: Instead of using a simple mathematical sum, this version uses a sophisticated gated unit (like a GRU) to update the coverage memory. This allows the model to learn its own complex strategy for tracking what has been translated.
In both cases, this coverage information is fed back into the attention mechanism at every step. If a word has already been "heavily attended to," the model is discouraged from focusing on it again, effectively pushing the "attention spotlight" toward untranslated words.
Datasets and Evaluation
The researchers evaluated their models on a Chinese-to-English translation task:
- Training: 1.25 million sentence pairs from LDC corpora.
- Evaluation: NIST 2002 (development set) and NIST 2005, 2006, and 2008 (test sets).
- Alignment: A specialized dataset of 900 manually aligned Chinese-English sentence pairs to test how well the model connects specific words across languages.
Key Results and Insights
The proposed NMT-COVERAGE model outperformed the baseline (GroundHog) across all metrics:
- Translation Quality: The NN-based model achieved an average BLEU score of 30.14, a significant jump over the baseline's 28.32 (+1.82 points).
- Alignment Accuracy: Alignment error rates (AER) dropped significantly, meaning the model became much better at "pointing" to the correct source word while translating.
- Subjective Improvement: Human evaluators found that the coverage model reduced under-translation errors from 25.0% down to 16.7%.
Beyond raw scores, the paper demonstrates that "old" ideas from statistical translation—like coverage and fertility—are still highly relevant in the era of deep learning. The coverage mechanism mitigates the "long sentence problem" by providing a structural "map" that prevents the model from getting lost in complex strings of text.
Conclusion
By introducing explicit memory into the attention mechanism, the coverage model effectively bridges the gap between the flexibility of neural networks and the structural rigor of statistical translation. While the NN-based model is more powerful due to its ability to learn non-linear relationships, both approaches represent a significant step forward in making NMT more reliable for real-world applications.
Summary updated