Abstract
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.
Introduction
Transformers have revolutionized artificial intelligence but have long been constrained by a "depth ceiling." Standard architectures are notoriously unstable as they get deeper, forcing researchers to choose between two sub-optimal Layer Normalization (LN) configurations: Post-LN, which offers high performance but extreme instability, and Pre-LN, which is stable but often results in lower performance due to gradient imbalances across layers.
The researchers identified that the root cause of this instability is the explosion of model updates at the start of training. These large updates trap the model in poor local optima, creating a jagged loss landscape that prevents convergence.
DeepNorm: Stable Scaling Through Theoretical Bounding
The core innovation of this work is DeepNorm, a new normalization function for residual connections. DeepNorm modifies the standard residual formula to: $$x_{l+1} = LN(\alpha x_l + G_l(x_l, \theta_l))$$ where $\alpha$ scales the residual connection and $\beta$ (within the sub-layer $G_l$) scales the weights. These parameters are mathematically derived to ensure that model updates remain bounded by a constant, regardless of depth. This allows DeepNet to achieve the high performance of Post-LN with the training stability of Pre-LN.
Key Insights: Depth vs. Width and Landscape Smoothing
A primary finding is that increasing model depth is a more parameter-efficient scaling strategy than increasing width. For example, a 200-layer DeepNet with 3.2B parameters outperformed a 48-layer state-of-the-art model with 12B parameters by 5 BLEU points on a massive multilingual translation task. This suggests that depth provides a more effective inductive bias for complex tasks.
Furthermore, visualization of the loss landscape explains why standard models fail. Vanilla Post-LN models exhibit a "mountainous," jagged loss surface at the start of training. DeepNorm effectively flattens this landscape, allowing the optimizer to find a smooth path to convergence.
Experimental Results
DeepNet was validated across a diverse set of benchmarks, demonstrating its robustness and scalability:
- 1,000-Layer Scaling: The authors successfully trained a Transformer with 1,000 layers (2,500 sub-layers) without the typical instability issues.
- Multilingual Translation: On the OPUS-100 benchmark (7,482 directions), DeepNet achieved a 4.4 BLEU point gain over the 48-layer baseline.
- Vision Pre-training: Integrated into the BEiT framework, DeepNet improved top-1 accuracy on ImageNet and boosted semantic segmentation performance on ADE20k by up to 1.1% mIoU.
- Stability: DeepNet allowed for larger learning rates and eliminated the need for the complex "warm-up" schedules typically required for deep Transformers.
Conclusion and Future Directions
DeepNet demonstrates that scaling depth is a powerful and efficient direction for Transformer models. By theoretically bounding updates, it enables unprecedented depth while maintaining stability. While deeper models are more parameter-efficient, they do increase sequential inference costs. Future work will explore applying DeepNet to other complex domains, such as protein structure prediction.
Summary updated