Coverage-based Neural Machine Translation

Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, Hang Li

Abstract

Standard Neural Machine Translation (NMT) models often "forget" which parts of a source sentence they have already translated, leading to repetitive or missing words. This paper introduces a coverage mechanism—a digital checklist that tracks translation history—to ensure every source word is accounted for, significantly improving both translation accuracy and alignment.

The Problem: Over-translation and Under-translation

When this research was conducted, NMT was rapidly overtaking traditional Statistical Machine Translation (SMT). However, it lacked a critical feature of SMT: a "coverage" mechanism. In older SMT systems, the decoder kept a strict record of which source words were "covered." In contrast, the then-state-of-the-art attention-based NMT models treated each step of translation as somewhat independent. The attention mechanism would "look" at the source sentence to decide what to translate next, but it didn't have a reliable memory of what it had already done.

This led to two major issues:

  1. Over-translation: The model gets stuck in a loop, translating the same source word multiple times (e.g., "close close").
  2. Under-translation: The model ignores certain words entirely, leaving parts of the original message out of the final translation.

Methods: Implementing Coverage and Fertility

The authors propose adding a coverage vector to the NMT architecture. This vector summarizes the attention history for each source word. They explore two primary ways to implement this:

  • Linguistic Coverage Model: This is a "soft" version of the old SMT approach. It tracks the sum of past attention weights for each word and introduces Fertility—predicting how many target words a specific source word should produce based on its context.
  • Neural Network (NN) Based Coverage Model: Instead of using a simple mathematical sum, this version uses a sophisticated gated unit (like a GRU) to update the coverage memory. This allows the model to learn its own complex strategy for tracking what has been translated.
The NMT architecture enhanced with a coverage mechanism.

In both cases, this coverage information is fed back into the attention mechanism at every step. If a word has already been "heavily attended to," the model is discouraged from focusing on it again, effectively pushing the "attention spotlight" toward untranslated words.

Datasets and Evaluation

The researchers evaluated their models on a Chinese-to-English translation task:

  • Training: 1.25 million sentence pairs from LDC corpora.
  • Evaluation: NIST 2002 (development set) and NIST 2005, 2006, and 2008 (test sets).
  • Alignment: A specialized dataset of 900 manually aligned Chinese-English sentence pairs to test how well the model connects specific words across languages.

Key Results and Insights

The proposed NMT-COVERAGE model outperformed the baseline (GroundHog) across all metrics:

  • Translation Quality: The NN-based model achieved an average BLEU score of 30.14, a significant jump over the baseline's 28.32 (+1.82 points).
  • Alignment Accuracy: Alignment error rates (AER) dropped significantly, meaning the model became much better at "pointing" to the correct source word while translating.
  • Subjective Improvement: Human evaluators found that the coverage model reduced under-translation errors from 25.0% down to 16.7%.
Attention heatmap demonstrating precise alignment between source and target sequences.

Beyond raw scores, the paper demonstrates that "old" ideas from statistical translation—like coverage and fertility—are still highly relevant in the era of deep learning. The coverage mechanism mitigates the "long sentence problem" by providing a structural "map" that prevents the model from getting lost in complex strings of text.

Conclusion

By introducing explicit memory into the attention mechanism, the coverage model effectively bridges the gap between the flexibility of neural networks and the structural rigor of statistical translation. While the NN-based model is more powerful due to its ability to learn non-linear relationships, both approaches represent a significant step forward in making NMT more reliable for real-world applications.

Summary updated

BibTeX
@inproceedings{tu2016coverage,
  author = {Zhaopeng Tu and Zhengdong Lu and Yang Liu and Xiaohua Liu and Hang Li},
  title = {{Coverage-based Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-1008},
}
Figure