Introduction
In the late 2010s, Neural Machine Translation (NMT) underwent a paradigm shift. Recurrent Neural Networks (RNNs), once the standard, were rapidly superseded by Convolutional (CNN) and Self-Attentional (Transformer) architectures. The theoretical justification for this transition often centered on "path length": while RNNs require $n$ sequential steps to connect words $n$ positions apart, Transformers achieve this in $O(1)$ time. However, this paper argues that the success of Transformers is not primarily due to bridging long-distance dependencies better than RNNs, but rather their superior ability to extract semantic features.
Methodology and Architectures
To investigate these claims, the authors conducted a head-to-head comparison of three primary architectures: RNNS2S (LSTM-based), ConvS2S (CNN-based), and the Transformer.
Beyond standard BLEU scores, the study employed contrastive evaluation. Models were tasked with assigning higher probability to a correct target sentence over one containing a specific linguistic error, such as subject-verb agreement or incorrect word sense. Training was conducted on the WMT17 English-German dataset, with targeted testing using Lingeval97 (for agreement) and ContraWSD (for word sense disambiguation).
Debunking the Path Length Myth
A core finding of the study is that Transformers and CNNs do not significantly outperform deep RNNs on long-range subject-verb agreement. Empirical results show that deep RNNs are remarkably robust as the distance between a subject and its verb increases.
As shown in the figure above, while the Transformer maintains a slight edge (97.1% vs. 96.3% for a bi-deep RNN), the performance gap does not widen with distance. This suggests that the theoretical "vanishing gradient" or "forgetting" issues in RNNs are less impactful in modern deep NMT implementations than previously assumed. Conversely, CNNs performed the worst on long-range tasks because their receptive field is strictly limited by kernel size and layer depth.
Transformers as Semantic Feature Extractors
If long-range syntax isn't the differentiator, what is? The authors found that the Transformer's true advantage lies in Word Sense Disambiguation (WSD). In DE→EN tasks, the Transformer achieved 90.3% accuracy, significantly outperforming RNNS2S (84.0%) and ConvS2S (82.3%). This indicates that self-attention is a more powerful mechanism for extracting the nuanced semantic features required to resolve lexical ambiguity.
Furthermore, the study highlights the necessity of multi-head attention. Reducing the number of heads caused a collapse in the model's ability to handle distant dependencies, suggesting that multiple heads are required to simultaneously process local syntactic structures and global semantic context.
Conclusion
While Transformers consistently lead in BLEU scores, this research clarifies that their dominance is not rooted in a superior "memory" for long-range syntax compared to optimized RNNs. Instead, their strength lies in superior semantic feature extraction and the parallel processing capabilities of multi-head attention. Future work remains to explore how these architectural trade-offs behave in non-translation tasks and how different attention heads specifically partition linguistic labor.
Summary updated