Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

Gongbo Tang, Mathias Müller, Annette Rios, Rico Sennrich

Introduction

In the late 2010s, Neural Machine Translation (NMT) underwent a paradigm shift. Recurrent Neural Networks (RNNs), once the standard, were rapidly superseded by Convolutional (CNN) and Self-Attentional (Transformer) architectures. The theoretical justification for this transition often centered on "path length": while RNNs require $n$ sequential steps to connect words $n$ positions apart, Transformers achieve this in $O(1)$ time. However, this paper argues that the success of Transformers is not primarily due to bridging long-distance dependencies better than RNNs, but rather their superior ability to extract semantic features.

Methodology and Architectures

To investigate these claims, the authors conducted a head-to-head comparison of three primary architectures: RNNS2S (LSTM-based), ConvS2S (CNN-based), and the Transformer.

Comparison of RNN, CNN, and Self-attention architectures.

Beyond standard BLEU scores, the study employed contrastive evaluation. Models were tasked with assigning higher probability to a correct target sentence over one containing a specific linguistic error, such as subject-verb agreement or incorrect word sense. Training was conducted on the WMT17 English-German dataset, with targeted testing using Lingeval97 (for agreement) and ContraWSD (for word sense disambiguation).

Debunking the Path Length Myth

A core finding of the study is that Transformers and CNNs do not significantly outperform deep RNNs on long-range subject-verb agreement. Empirical results show that deep RNNs are remarkably robust as the distance between a subject and its verb increases.

Accuracy on subject-verb agreement as a function of distance between dependencies.

As shown in the figure above, while the Transformer maintains a slight edge (97.1% vs. 96.3% for a bi-deep RNN), the performance gap does not widen with distance. This suggests that the theoretical "vanishing gradient" or "forgetting" issues in RNNs are less impactful in modern deep NMT implementations than previously assumed. Conversely, CNNs performed the worst on long-range tasks because their receptive field is strictly limited by kernel size and layer depth.

Transformers as Semantic Feature Extractors

If long-range syntax isn't the differentiator, what is? The authors found that the Transformer's true advantage lies in Word Sense Disambiguation (WSD). In DE→EN tasks, the Transformer achieved 90.3% accuracy, significantly outperforming RNNS2S (84.0%) and ConvS2S (82.3%). This indicates that self-attention is a more powerful mechanism for extracting the nuanced semantic features required to resolve lexical ambiguity.

Furthermore, the study highlights the necessity of multi-head attention. Reducing the number of heads caused a collapse in the model's ability to handle distant dependencies, suggesting that multiple heads are required to simultaneously process local syntactic structures and global semantic context.

Conclusion

While Transformers consistently lead in BLEU scores, this research clarifies that their dominance is not rooted in a superior "memory" for long-range syntax compared to optimized RNNs. Instead, their strength lies in superior semantic feature extraction and the parallel processing capabilities of multi-head attention. Future work remains to explore how these architectural trade-offs behave in non-translation tasks and how different attention heads specifically partition linguistic labor.

Summary updated

Abstract

Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.
BibTeX
@inproceedings{tang2018why,
  author = {Gongbo Tang and Mathias Müller and Annette Rios and Rico Sennrich},
  title = {{Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1458},
}
Figure