Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention

Biao Zhang, Ivan Titov, Rico Sennrich

While the "bigger is better" trend dominates modern AI, simply stacking layers in a Transformer isn't as easy as it looks. In Neural Machine Translation (NMT), the standard Transformer often hits a wall at just 6 layers; try to go deeper, and the model frequently fails to converge.

In their 2019 paper, "Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention," Biao Zhang and colleagues from the University of Edinburgh and Amsterdam identify exactly why deep Transformers break and provide two elegant solutions to scale them up without sacrificing speed.

The Gradient Vanishing Problem

In theory, deeper models should have higher capacity and better performance. However, vanilla Transformers suffer from gradient vanishing as they get deeper.

The authors' analysis points to a structural culprit: the interaction between Residual Connections (RC) and Layer Normalization (LN). While residual connections are meant to help gradients flow, the large output variance they produce causes the subsequent Layer Norm to "squash" the error signal too aggressively. By the time the gradient reaches the lower layers of a 12- or 18-layer model, it has effectively disappeared, leading to poor training convergence.

Gradient norm across layers for standard vs. DS-Init Transformers in both Encoder (a) and Decoder (b).

Proposed Methods

The researchers proposed two primary interventions to stabilize training and maintain efficiency:

1. Depth-Scaled Initialization (DS-Init)

Instead of using standard uniform initialization for all layers, DS-Init scales down the variance of parameters in the $l$-th layer by a factor of $1/\sqrt{l}$. By reducing the parameter variance in higher layers, the output variance of the residual connections is narrowed. This prevents Layer Norm from over-correcting, allowing gradients to flow back to the earlier layers more effectively, as seen in the stabilized gradient norms in Figure 1.

2. Merged Attention (MAtt)

To combat the slowdown caused by adding layers, the authors redesigned the decoder block. Usually, a decoder has two separate attention sublayers: self-attention and encoder-decoder (cross) attention. MAtt replaces these with a single sublayer that combines a simplified average-based self-attention (AAN) with the cross-attention. This reduces the "depth" of each individual block and allows for parallel computation.

Architecture comparison between standard Self-Attention, Average Attention Networks (AAN), and the proposed Merged Attention (MAtt).

Key Results and Insights

The authors evaluated their methods on several competitive NMT benchmarks, including WMT14 (En-De, En-Fr), IWSLT14 (De-En), and WMT18 (En-Fi, Zh-En). Their findings highlight the effectiveness of these structural and initialization changes:

  • Successful Scaling: While vanilla Transformers failed to train effectively beyond 12 layers, the authors successfully trained models up to 30 layers deep, showing continued gains.
  • Better Translation: 12-layer models using DS-Init and MAtt outperformed 6-layer baselines by an average of +1.1 BLEU across five tasks. On WMT14 En-De, their "Big" 20-layer model achieved 28.67 BLEU.
  • Speed Efficiency: Thanks to MAtt, a 12-layer "deep" model was able to match the decoding speed of a standard 6-layer baseline. In some configurations, MAtt improved decoding speed by 50% compared to a standard deep implementation.
  • Initialization as Architecture: DS-Init proves that you don't always need to change the math of the model (like adding complex gating) to fix convergence; sometimes, just changing the starting point of the weights is enough to stabilize the entire training process.

Limitations and Future Work

Despite the success, the authors noted diminishing returns; performance gains started to plateau after 12–20 layers, suggesting other structural bottlenecks might exist beyond just gradient flow. Additionally, while MAtt helps with inference speed, training deep models still requires significantly more wall-clock time and memory than shallow ones. Future work may explore these techniques in other sequence-to-sequence tasks like summarization.

Summary updated

BibTeX
@inproceedings{zhang2019improving,
  author = {Biao Zhang and Ivan Titov and Rico Sennrich},
  title = {{Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ir-PffL8Thj},
}
Figure