Abstract
Standard attention-based Neural Machine Translation (NMT) models often suffer from alignment errors, such as word skipping or repetition, due to a lack of explicit distortion and fertility modeling. This paper introduces Recurrent Attention (RecAtt) and a Conditioned Decoder (CondDec) to provide the model with implicit memory of its alignment history. Experimental results on Chinese-to-English translation tasks show a 2.1 BLEU point improvement over baseline systems, producing cleaner alignments and more fluent translations.
The Challenge: Alignment in NMT
In the early development of NMT, the attention mechanism was a major breakthrough, allowing decoders to dynamically focus on different parts of a source sentence. However, these models frequently exhibit alignment errors. Unlike traditional Statistical Machine Translation (SMT) systems, standard NMT attention mechanisms lack explicit mechanisms for:
- Distortion: Managing complex word re-ordering across languages.
- Fertility: Tracking how many target words a single source word should generate.
Without these, models often produce repetitive phrasing or skip entire segments of the source text because the attention mechanism is "stateless" and unaware of its previous focus.
Proposed Solutions: Recurrent Attention and Conditioned Decoders
The authors propose three architectural enhancements to address these limitations:
Recurrent Attention (RecAtt and RNNAtt)
Standard attention relies solely on the current decoder state. RecAtt introduces a feedback loop where the previous context vector is fed back into the attention unit, providing a "short-term memory" of the alignment path. RNNAtt extends this by using a full Recurrent Neural Network (RNN) within the attention unit to maintain a "long-term memory."
Conditioned Decoder (CondDec)
To model "fertility," the CondDec uses a condition vector initialized with source sentence information. This vector "decays" as information is extracted by the decoder. Two auxiliary costs—Step-decay and Left-over—are added during training to ensure the model utilizes all source information exactly once, effectively acting as a translation "checklist."
Experimental Setup
The models were evaluated on the NIST Chinese-to-English translation task using:
- Training: 0.5 million sentence pairs.
- Validation/Testing: NIST 2003 and 2005 datasets.
- Baselines: RNNSearch (standard attention), InputFeed, and hierarchical phrase-based SMT.
Results and Analysis
The proposed modifications yielded significant improvements:
- RecAtt achieved the highest performance with 33.14 BLEU points, a 2.12 point gain over the RNNSearch baseline.
- CondDec successfully mitigated coverage and repetition issues, outperforming the baseline by 1.19 BLEU points.
- Qualitative analysis of alignment heatmaps showed that RecAtt and CondDec produced much cleaner, diagonal alignments compared to the noisy outputs of baseline models.
While RNNAtt was theoretically more powerful, it slightly underperformed compared to RecAtt, likely due to vanishing gradient issues in the longer backpropagation path through the attention RNN.
Conclusion
By incorporating implicit distortion and fertility models through recurrence and conditioned decoding, the authors significantly improved the robustness of NMT alignment. These enhancements allow the model to maintain a better "memory" of the translation process, leading to higher quality and more reliable machine translations.
Summary updated