Jointly Learning to Align and Translate with Transformer Models

Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias PaulikEMNLP 2019 · DOI

TL;DR

This paper introduces a novel multi-task learning approach for Transformer models that simultaneously learns to translate and align words between languages. By leveraging attention probabilities and a carefully designed loss function, they achieve state-of-the-art word alignment accuracy without sacrificing translation quality, even outperforming traditional statistical methods.

What Problem Does This Paper Address?

Machine translation (MT) has seen tremendous progress with neural networks, particularly the Transformer architecture. However, a related task, word alignment (determining which words in a source sentence correspond to which words in a target sentence), has largely remained the domain of older statistical methods like GIZA++. While Transformers use attention mechanisms that can be interpreted as alignments, these attention probabilities don't always directly translate to high-quality discrete alignments. Extracting these alignments is crucial for tasks like building bilingual lexicons, improving translation of low-frequency words, and preserving document annotations. Existing methods for aligning with Transformers either rely on external statistical tools or struggle to match their performance. This paper aims to bridge this gap by enabling Transformers to learn accurate word alignments jointly with translation.

Methods

The core of this work is a multi-task learning framework for Transformer models. Here's a breakdown of their key techniques:

  • Leveraging Attention Probabilities: Instead of treating attention as a black box, the authors extract discrete alignments from the attention probabilities learned during regular NMT training.
  • Multi-Task Loss Function: They combine the standard negative log-likelihood (NLL) loss for translation with a new alignment loss. This alignment loss supervises one specific attention head within the Transformer's decoder.
  • Self-Training for Alignment Supervision: To avoid relying solely on external statistical aligners, they first generate "seed" alignments by averaging attention probabilities from a regular NMT model (specifically, from the penultimate layer, which they found to be most effective). These generated alignments are then used as labels to train their multi-task model.
  • Contextual Conditioning: A key challenge is that standard Transformer decoders are auto-regressive, meaning they only consider past target tokens for translation. This is limiting for alignment, which might benefit from seeing the entire target sentence. The authors address this by conditioning the translation loss (Lt) on past target tokens (preserving auto-regressivity) but conditioning the alignment loss (L'a) on the full target sentence context. This is achieved by running two forward passes of the decoder.
  • Optional GIZA++ Supervision: For even higher alignment accuracy, they also explore a setup where they use alignments from the established GIZA++ toolkit as supervision for their alignment head.

Datasets

The paper uses several standard datasets for their experiments:

  • Romanian-English (RoEn), English-French (EnFr), and German-English (DeEn): These datasets are used for evaluating the alignment task. They include training data and test data with gold (human-annotated) alignments. The RoEn dataset is augmented with the Europarl v7 corpus.
  • WMT'18 English-German news translation task: This dataset is used for evaluating the joint align-and-translate task. They utilize a combination of corpora including Europarl v7, Common Crawl, News Commentary v13, and Rapid corpus.

Key Insights and Ideas

  • Penultimate Layer Attention is Key: The authors discovered that attention probabilities from the penultimate layer of the Transformer decoder tend to learn alignments more effectively than other layers, outperforming naive averaging across all layers.
  • Joint Learning is Beneficial: The multi-task approach, by jointly optimizing for translation and alignment, shows that learning to translate can actually help produce better alignments, and vice-versa.
  • Full Target Context Matters for Alignment: Providing the alignment head with access to the entire target sentence (not just past tokens) significantly improves alignment accuracy, suggesting that context is crucial for disambiguating word correspondences.
  • Self-Training is Viable: The ability to generate reasonable alignment supervision from a standard NMT model (layer average baseline) is a significant contribution, reducing reliance on external, potentially complex, statistical tools.
  • Pronoun Alignment Behavior: In their analysis, they observed that their model sometimes aligns pronouns with nouns, which they hypothesize is due to NMT models' ability to encode contextual information, a capability beneficial for word sense disambiguation and, by extension, alignment.

Key Results

  • Alignment Accuracy: Their multi-task approach, especially when supervised with GIZA++ alignments, significantly outperforms GIZA++ (BPE-based) in terms of Alignment Error Rate (AER). For example, on the German-English dataset, their best model achieved an AER of 16.4%, compared to GIZA++ (BPE-based) at 19.6%.
  • Translation Quality: Crucially, these improvements in alignment accuracy come without sacrificing translation performance. They report comparable BLEU scores to strong baselines.
  • Outperforming Previous Neural Methods: Their approach also surpasses previous neural methods like Zenkel et al. (2019) in alignment accuracy.
  • Layer-wise Attention Analysis: Their analysis of attention across layers showed a clear trend of alignment quality improving in higher layers, with the penultimate layer being particularly effective.

Limitations and Future Work

  • Two Training Runs: The current self-training approach requires two distinct training runs: one to generate seed alignments and another for the multi-task training. The authors suggest investigating a single-run training method that alternates between alignment extraction and model training.
  • Alignment Validation Set: For the WMT'18 English-German task, they note the lack of a dedicated alignment validation set, making it difficult to pick the optimal checkpoint for alignment supervision.
  • Pronoun Alignment: While they observe interesting pronoun-noun alignment patterns, further investigation into why this occurs and its implications could be a future direction.
  • Hyperparameter Tuning: The alignment loss weight (λ) was tuned using grid search, and finding optimal hyperparameters for different language pairs or tasks might require further exploration.

Summary updated

BibTeX
@inproceedings{garg2019jointly,
  author = {Sarthak Garg and Stephan Peitz and Udhyakumar Nallasamy and Matthias Paulik},
  title = {{Jointly Learning to Align and Translate with Transformer Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1453},
}
Figure