Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model

Shi Feng, Shujie Liu, Mu Li, Ming ZhouarXiv 2016 · arXiv

Abstract

Standard attention-based Neural Machine Translation (NMT) models often suffer from alignment errors, such as word skipping or repetition, due to a lack of explicit distortion and fertility modeling. This paper introduces Recurrent Attention (RecAtt) and a Conditioned Decoder (CondDec) to provide the model with implicit memory of its alignment history. Experimental results on Chinese-to-English translation tasks show a 2.1 BLEU point improvement over baseline systems, producing cleaner alignments and more fluent translations.

The Challenge: Alignment in NMT

In the early development of NMT, the attention mechanism was a major breakthrough, allowing decoders to dynamically focus on different parts of a source sentence. However, these models frequently exhibit alignment errors. Unlike traditional Statistical Machine Translation (SMT) systems, standard NMT attention mechanisms lack explicit mechanisms for:

  1. Distortion: Managing complex word re-ordering across languages.
  2. Fertility: Tracking how many target words a single source word should generate.

Without these, models often produce repetitive phrasing or skip entire segments of the source text because the attention mechanism is "stateless" and unaware of its previous focus.

Figure 4: An alignment heatmap illustrating common errors in baseline NMT models, such as repetitive attention on specific source phrases.

Proposed Solutions: Recurrent Attention and Conditioned Decoders

The authors propose three architectural enhancements to address these limitations:

Recurrent Attention (RecAtt and RNNAtt)

Standard attention relies solely on the current decoder state. RecAtt introduces a feedback loop where the previous context vector is fed back into the attention unit, providing a "short-term memory" of the alignment path. RNNAtt extends this by using a full Recurrent Neural Network (RNN) within the attention unit to maintain a "long-term memory."

Figure 5: The proposed Recurrent Attention (RecAtt) architecture, where the previous context vector is fed back into the attention mechanism.

Conditioned Decoder (CondDec)

To model "fertility," the CondDec uses a condition vector initialized with source sentence information. This vector "decays" as information is extracted by the decoder. Two auxiliary costs—Step-decay and Left-over—are added during training to ensure the model utilizes all source information exactly once, effectively acting as a translation "checklist."

Experimental Setup

The models were evaluated on the NIST Chinese-to-English translation task using:

  • Training: 0.5 million sentence pairs.
  • Validation/Testing: NIST 2003 and 2005 datasets.
  • Baselines: RNNSearch (standard attention), InputFeed, and hierarchical phrase-based SMT.

Results and Analysis

The proposed modifications yielded significant improvements:

  • RecAtt achieved the highest performance with 33.14 BLEU points, a 2.12 point gain over the RNNSearch baseline.
  • CondDec successfully mitigated coverage and repetition issues, outperforming the baseline by 1.19 BLEU points.
  • Qualitative analysis of alignment heatmaps showed that RecAtt and CondDec produced much cleaner, diagonal alignments compared to the noisy outputs of baseline models.

While RNNAtt was theoretically more powerful, it slightly underperformed compared to RecAtt, likely due to vanishing gradient issues in the longer backpropagation path through the attention RNN.

Conclusion

By incorporating implicit distortion and fertility models through recurrence and conditioned decoding, the authors significantly improved the robustness of NMT alignment. These enhancements allow the model to maintain a better "memory" of the translation process, leading to higher quality and more reliable machine translations.

Summary updated

BibTeX
@misc{feng2016implicit,
  author = {Shi Feng and Shujie Liu and Mu Li and Ming Zhou},
  title = {{Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model}},
  year = {2016},
  eprint = {1601.03317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1601.03317},
}
Figure