Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation

Shi Feng, Shujie Liu, Nan Yang, Mu Li, Ming Zhou, Kenny Q. ZhuCOLING 2016 · OpenReview

Abstract

In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence. However, without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality. In this paper, we propose two novel models to improve attention-based neural machine translation. We propose a recurrent attention mechanism as an implicit distortion model, and a fertility conditioned decoder as an implicit fertility model. We conduct experiments on large-scale Chinese–English translation tasks. The results show that our models significantly improve both the alignment and translation quality compared to the original attention mechanism and several other variations.

Introduction

While attention-based Neural Machine Translation (NMT) has revolutionized the field, it often lacks the explicit structural constraints found in traditional Statistical Machine Translation (SMT). Specifically, NMT models frequently struggle with distortion (word reordering) and fertility (the number of times a source word should be translated). Without these concepts, standard NMT systems often suffer from a "short-term memory" problem, leading to three primary failure modes: incorrect reordering, missing translations, and repeated translations.

Addressing Distortion and Fertility

This research bridges the gap between SMT and NMT by reintroducing these concepts through two complementary architectural extensions to the standard RNNSearch (attention) framework.

1. RECATT: Recurrent Attention

To handle distortion, the authors propose RECATT. In standard attention models, the alignment score is calculated independently at each step based only on the current decoder state and source hidden states. RECATT makes this process recurrent by passing the previous context vector—a summary of what was just attended to—into the next attention calculation. This "content-aware" approach gives the model a sense of its previous focus, allowing it to make more informed jumps to the next logical word in a sequence.

The RECATT architecture introduces a feedback loop (dashed line) from the context vector back into the alignment model.

2. FERTDEC: Fertility-Conditioned Decoder

To manage fertility, the authors introduce FERTDEC, which utilizes a coverage vector ($d_i$) that functions as a "to-do list" of untranslated source information. This vector is updated at each step via an "extract gate" to track which parts of the source sentence have been "consumed." To reinforce this, the training process includes two new costs: step-decay (to prevent over-translation in a single step) and left-over (to penalize the model if source material remains untranslated at the end of a sentence).

Experimental Results

The models were evaluated on a large-scale Chinese–English translation task using the NIST dataset. The combined model (RECATT + FERTDEC) demonstrated significant improvements over the baseline:

  • Translation Quality: The BLEU score increased from 31.02 to 33.76 (after UNK replacement).
  • Alignment Accuracy: The Alignment Error Rate (AER) was reduced from 44.13 to 42.11, indicating that the model's internal attention was more accurately tracking the source text.

Qualitative Analysis

Qualitative evidence highlights how these mechanisms fix common NMT errors. For instance, standard models often get stuck in loops, repeatedly translating the same phrase. FERTDEC successfully terminates these translations by tracking coverage, preventing the "looping" behavior seen in baseline models.

An example of a repeated translation error in a baseline model, where it gets stuck in a loop translating "night of the midnight of the night."

Conclusion

By reintroducing implicit models for distortion and fertility, the authors significantly improved the robustness of attention-based NMT. The results suggest that better internal alignment directly correlates with higher translation quality. Future work may explore more sophisticated ways to encode alignment history beyond the previous context vector to further refine distortion modeling.

Summary updated

Abstract

In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence. However, without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality. In this paper, we propose two novel models to improve attention-based neural machine translation. We propose a recurrent attention mechanism as an implicit distortion model, and a fertility conditioned decoder as an implicit fertility model. We conduct experiments on large-scale Chinese–English translation tasks. The results show that our models significantly improve both the alignment and translation quality compared to the original attention mechanism and several other variations.
BibTeX
@inproceedings{feng2016improving,
  author = {Shi Feng and Shujie Liu and Nan Yang and Mu Li and Ming Zhou and Kenny Q. Zhu},
  title = {{Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=4x2xpfKKK1},
}
Figure