Abstract
The general trend in NLP is towards increasing model capacity and performance via deeper neural networks. However, simply stacking more layers of the popular Transformer architecture for machine translation results in poor convergence and high computational overhead. Our empirical analysis suggests that convergence is poor due to gradient vanishing caused by the interaction between residual connections and layer normalization. We propose depth-scaled initialization (DS-Init), which decreases parameter variance at the initialization stage, and reduces output variance of residual connections so as to ease gradient back-propagation through normalization layers. To address computational cost, we propose a merged attention sublayer (MAtt) which combines a simplified average-based self-attention sublayer and the encoder-decoder attention sublayer on the decoder side. Results on WMT and IWSLT translation tasks show that deep Transformers with DS-Init and MAtt can substantially outperform their base counterpart (+1.1 BLEU on average for 12-layer models), while matching the decoding speed of the baseline model.
The Convergence and Computational Bottleneck
In computer vision, scaling neural networks to hundreds of layers reliably improves performance. However, in Natural Language Processing, the standard encoder-decoder Transformer architecture historically peaks at just 6 layers. Simply stacking more layers leads to poor convergence and significant computational overhead.
The authors trace this failure to vanishing gradients caused by the interaction between Residual Connections (RC) and Layer Normalization (LN). Furthermore, even when deep models converge, the computational cost of separate self-attention and cross-attention sublayers in the decoder makes inference painfully slow. This research addresses both the mathematical convergence issue and the practical computational bottleneck.
Proposed Methods
1. Depth-Scaled Initialization (DS-Init)
Instead of altering the model's architecture (e.g., moving to Pre-Norm), the authors propose changing the weight initialization. They scale down the variance of parameters in the $l$-th layer by a discount factor of $1/\sqrt{l}$. By forcing higher layers to have smaller parameter variances, the output variance of the residual connections is kept in check. This allows the error signal to back-propagate through the Layer Normalization steps without vanishing.
2. Merged Attention (MAtt)
To address the computational cost of deeper networks, the authors redesign the decoder. A standard Transformer decoder block has separate self-attention and encoder-decoder (cross) attention sublayers. MAtt replaces these with a single merged sublayer. It uses a highly simplified Average Attention Network (AAN) for self-attention and computes it in parallel with the encoder-decoder attention. This reduces the number of linear transformations and flattens the depth of each decoder block.
Key Insights
- The RC/LN Trap: Layer Normalization actively weakens the error signal during backpropagation, an effect exacerbated by the large output variance of standard Residual Connections.
- Decoder Sensitivity: Gradients are preserved better through self-attention than through encoder-decoder (cross) attention. Because cross-attention only back-propagates gradients to lower layers through the "query" representation, the decoder suffers from worse gradient vanishing than the encoder.
- Initialization over Architecture: Mathematically grounded initialization (DS-Init) is sufficient to preserve gradient norms across dozens of layers without requiring complex architectural routing.
Experimental Results
- Translation Quality: Deep Transformers (12 layers) with DS-Init and MAtt outperformed 6-layer baselines by an average of +1.1 BLEU across five translation tasks. On WMT14 En-De, a 20-layer model achieved 28.67 BLEU, compared to 27.59 for the baseline.
- Efficiency: The MAtt sublayer improves decoding speed by 50% and training speed by 10%, while using 9% fewer parameters.
- Speed Parity: When combining both techniques, a 12-layer deep model decodes at roughly the same speed as a standard 6-layer baseline, providing the performance of a deep model for the computational price of a shallow one.
Conclusion
While performance gains begin to plateau beyond 12 layers, DS-Init and MAtt successfully enable the training of Transformers up to 30 layers deep. These techniques demonstrate that the limitations of Transformer depth are not inherent to the attention mechanism itself, but rather to initialization and sublayer redundancy.
Summary updated