DeepNet: Scaling Transformers to 1,000 Layers

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei

Abstract

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

Introduction

Transformers have revolutionized artificial intelligence but have long been constrained by a "depth ceiling." Standard architectures are notoriously unstable as they get deeper, forcing researchers to choose between two sub-optimal Layer Normalization (LN) configurations: Post-LN, which offers high performance but extreme instability, and Pre-LN, which is stable but often results in lower performance due to gradient imbalances across layers.

The researchers identified that the root cause of this instability is the explosion of model updates at the start of training. These large updates trap the model in poor local optima, creating a jagged loss landscape that prevents convergence.

DeepNorm: Stable Scaling Through Theoretical Bounding

The core innovation of this work is DeepNorm, a new normalization function for residual connections. DeepNorm modifies the standard residual formula to: $$x_{l+1} = LN(\alpha x_l + G_l(x_l, \theta_l))$$ where $\alpha$ scales the residual connection and $\beta$ (within the sub-layer $G_l$) scales the weights. These parameters are mathematically derived to ensure that model updates remain bounded by a constant, regardless of depth. This allows DeepNet to achieve the high performance of Post-LN with the training stability of Pre-LN.

Gradient norm across layers for Post-LN vs DeepNet (Post-LN-init), demonstrating significantly improved stability in deep architectures.

Key Insights: Depth vs. Width and Landscape Smoothing

A primary finding is that increasing model depth is a more parameter-efficient scaling strategy than increasing width. For example, a 200-layer DeepNet with 3.2B parameters outperformed a 48-layer state-of-the-art model with 12B parameters by 5 BLEU points on a massive multilingual translation task. This suggests that depth provides a more effective inductive bias for complex tasks.

Furthermore, visualization of the loss landscape explains why standard models fail. Vanilla Post-LN models exhibit a "mountainous," jagged loss surface at the start of training. DeepNorm effectively flattens this landscape, allowing the optimizer to find a smooth path to convergence.

Comparison of loss landscapes: (a) Post-LN Transformers show a jagged, unstable surface, while (b) DeepNet provides a smooth, optimizable landscape.

Experimental Results

DeepNet was validated across a diverse set of benchmarks, demonstrating its robustness and scalability:

  • 1,000-Layer Scaling: The authors successfully trained a Transformer with 1,000 layers (2,500 sub-layers) without the typical instability issues.
  • Multilingual Translation: On the OPUS-100 benchmark (7,482 directions), DeepNet achieved a 4.4 BLEU point gain over the 48-layer baseline.
  • Vision Pre-training: Integrated into the BEiT framework, DeepNet improved top-1 accuracy on ImageNet and boosted semantic segmentation performance on ADE20k by up to 1.1% mIoU.
  • Stability: DeepNet allowed for larger learning rates and eliminated the need for the complex "warm-up" schedules typically required for deep Transformers.

Conclusion and Future Directions

DeepNet demonstrates that scaling depth is a powerful and efficient direction for Transformer models. By theoretically bounding updates, it enables unprecedented depth while maintaining stability. While deeper models are more parameter-efficient, they do increase sequential inference costs. Future work will explore applying DeepNet to other complex domains, such as protein structure prediction.

Summary updated

Abstract

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.
BibTeX
@article{wang2024deepnet,
  author = {Hongyu Wang and Shuming Ma and Li Dong and Shaohan Huang and Dongdong Zhang and Furu Wei},
  title = {{DeepNet: Scaling Transformers to 1,000 Layers}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=DAMMgOFn5M},
}
Figure