Generating Alignments Using Target Foresight in Attention-Based Neural Machine Translation

Jan-Thorsten Peter, Arne Nix, Hermann NeyPrague Bull. Math. Linguistics 2017 · OpenReview

Introduction

In machine translation, "alignment" refers to the task of identifying correspondences between words in a source sentence and their counterparts in a target sentence. While traditional statistical models like GIZA++ were designed specifically for this task, modern Neural Machine Translation (NMT) relies on an attention mechanism to dynamically focus on relevant source words during decoding.

However, standard NMT attention often produces "blurry" or inaccurate alignments. This occurs because the attention mechanism calculates weights based on the previous decoder state and source representations, without knowledge of the specific target word it is currently attempting to generate. This paper introduces Target Foresight (TF), a modification that provides the attention mechanism with a "sneak peek" at the target word to produce high-precision word mappings.

Methodology: Target Foresight and Guided Alignment

The core innovation of this work is the Target Foresight (TF) mechanism. In a standard NMT architecture, the attention energy is a function of the encoder states and the previous decoder state. The authors extend this by incorporating the embedding of the current target word ($e_i$) directly into the attention calculation.

Architecture of the NMT model with Target Foresight. The blue dashed lines indicate the flow of information from the target word back into the attention mechanism.

Overcoming the "Cheating" Problem

Initial experiments revealed that neural networks are prone to "cheating" when given the target word. Rather than learning to align, the model used the attention weights as a high-bandwidth channel to pass the target word's identity directly to the output layer, resulting in poor alignments. To counteract this, the authors combined TF with Guided Alignment (GA). This approach uses a secondary loss function to penalize the model if its attention weights deviate from a reference alignment (typically provided by a tool like GIZA++). This constraint ensures that the "foresight" information is used specifically to sharpen the mapping between source and target words.

Experimental Setup

The models were trained on the Europarl German-to-English corpus (~1.2 million sentence pairs). Alignment quality was evaluated using a gold-standard set of 504 manually aligned sentence pairs. To assess translation performance, the authors used the IWSLT 2013 German→English task, measuring success via BLEU scores.

Results and Discussion

The combination of Target Foresight and Guided Alignment yielded a massive improvement in alignment accuracy. The baseline NMT system's Alignment Error Rate (AER) of 38.1% was slashed to 19.0% when using TF + GA. Notably, this neural approach outperformed traditional gold-standard tools, including GIZA++ (21.0% AER) and the BerkeleyAligner (20.5% AER).

Comparison of alignment heatmaps. The TF + GA approach produces significantly sharper and more accurate alignments compared to the blurry baseline and the statistical GIZA++ model.

Beyond alignment metrics, the high-quality neural alignments also improved translation quality. When these alignments were used to guide the training of a standard NMT model, the authors observed a 2.8 point BLEU score improvement over the baseline and a 0.4 point improvement over systems guided by traditional statistical alignments.

Conclusion

Target Foresight demonstrates that providing future context to the attention mechanism can dramatically improve the interpretability and accuracy of NMT alignments. While the current implementation relies on external tools for initial guidance, it proves that neural models can surpass traditional statistical methods in word alignment. Future research will focus on achieving these sharp alignments without the need for external "gold" labels during training.

Summary updated

BibTeX
@inproceedings{peter2017generating,
  author = {Jan-Thorsten Peter and Arne Nix and Hermann Ney},
  title = {{Generating Alignments Using Target Foresight in Attention-Based Neural Machine Translation}},
  booktitle = {Prague Bull. Math. Linguistics},
  year = {2017},
  url = {https://openreview.net/forum?id=-hYZR_Z4tr},
}
Figure