Abstract
In neural machine translation, the attention mechanism facilitates the translation process by producing a soft alignment between the source sentence and the target sentence. However, without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality. In this paper, we propose two novel models to improve attention-based neural machine translation. We propose a recurrent attention mechanism as an implicit distortion model, and a fertility conditioned decoder as an implicit fertility model. We conduct experiments on large-scale Chinese–English translation tasks. The results show that our models significantly improve both the alignment and translation quality compared to the original attention mechanism and several other variations.
Introduction
While attention-based Neural Machine Translation (NMT) has revolutionized the field, it often lacks the explicit structural constraints found in traditional Statistical Machine Translation (SMT). Specifically, NMT models frequently struggle with distortion (word reordering) and fertility (the number of times a source word should be translated). Without these concepts, standard NMT systems often suffer from a "short-term memory" problem, leading to three primary failure modes: incorrect reordering, missing translations, and repeated translations.
Addressing Distortion and Fertility
This research bridges the gap between SMT and NMT by reintroducing these concepts through two complementary architectural extensions to the standard RNNSearch (attention) framework.
1. RECATT: Recurrent Attention
To handle distortion, the authors propose RECATT. In standard attention models, the alignment score is calculated independently at each step based only on the current decoder state and source hidden states. RECATT makes this process recurrent by passing the previous context vector—a summary of what was just attended to—into the next attention calculation. This "content-aware" approach gives the model a sense of its previous focus, allowing it to make more informed jumps to the next logical word in a sequence.
2. FERTDEC: Fertility-Conditioned Decoder
To manage fertility, the authors introduce FERTDEC, which utilizes a coverage vector ($d_i$) that functions as a "to-do list" of untranslated source information. This vector is updated at each step via an "extract gate" to track which parts of the source sentence have been "consumed." To reinforce this, the training process includes two new costs: step-decay (to prevent over-translation in a single step) and left-over (to penalize the model if source material remains untranslated at the end of a sentence).
Experimental Results
The models were evaluated on a large-scale Chinese–English translation task using the NIST dataset. The combined model (RECATT + FERTDEC) demonstrated significant improvements over the baseline:
- Translation Quality: The BLEU score increased from 31.02 to 33.76 (after UNK replacement).
- Alignment Accuracy: The Alignment Error Rate (AER) was reduced from 44.13 to 42.11, indicating that the model's internal attention was more accurately tracking the source text.
Qualitative Analysis
Qualitative evidence highlights how these mechanisms fix common NMT errors. For instance, standard models often get stuck in loops, repeatedly translating the same phrase. FERTDEC successfully terminates these translations by tracking coverage, preventing the "looping" behavior seen in baseline models.
Conclusion
By reintroducing implicit models for distortion and fertility, the authors significantly improved the robustness of attention-based NMT. The results suggest that better internal alignment directly correlates with higher translation quality. Future work may explore more sophisticated ways to encode alignment history beyond the previous context vector to further refine distortion modeling.
Summary updated