Abstract
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.
Introduction
DeepNet introduces DEEPNORM, a simple yet mathematically grounded normalization and initialization scheme that enables Transformers to scale to an unprecedented 1,000 layers. By bounding model updates during training, DeepNet achieves the high performance of Post-LN architectures with the stability of Pre-LN, allowing "deep and narrow" models to significantly outperform much larger "shallow and wide" counterparts.
The Challenge of Depth
In the race to scale Large Language Models (LLMs), researchers have primarily focused on increasing parameter counts (width). However, increasing depth—the number of sequential layers—has remained notoriously difficult due to training instability.
Standard Transformers typically use one of two setups:
- Post-LN: Normalization happens after the residual addition. It generally performs better but is prone to exploding gradients and updates, making it nearly impossible to train at great depths.
- Pre-LN: Normalization happens before the sub-layer. It is much more stable to train but suffers from gradient vanishing issues in the bottom layers, leading to a performance plateau as depth increases.
The authors of DeepNet sought to find the "best of both worlds": a method that provides the stability of Pre-LN while retaining the superior performance potential of Post-LN at extreme scales.
DeepNorm: Stable Scaling
The core contribution is DEEPNORM, a new normalization function paired with a theoretically derived initialization scheme.
1. The DEEPNORM Function
Instead of the standard residual connection ($x + f(x)$), DeepNet scales the residual connection by a constant $\alpha$ before applying LayerNorm: $$x_{l+1} = \text{LayerNorm}(\alpha x_l + f(x_l))$$
2. Scaled Initialization
The weights ($\theta$) within the sub-layers (like feed-forward networks and attention projections) are scaled by a constant $\beta$ during initialization.
3. Theoretical Grounding
The "magic" lies in how $\alpha$ and $\beta$ are calculated. The authors performed an in-depth theoretical analysis to ensure that the expected magnitude of model updates remains bounded by a constant. These values change depending on the architecture:
- Encoder-only: $\alpha = (2N)^{1/4}$, $\beta = (8N)^{-1/4}$ (where $N$ is the number of layers).
- Encoder-Decoder: Different constants are derived for the encoder and decoder to account for the added complexity of cross-attention.
This approach requires only a few lines of code change but fundamentally changes the optimization landscape, preventing the model from falling into "spurious local optima" at the start of training.
Results and Efficiency
The authors successfully trained a Transformer with 1,000 layers (comprising 2,500 sub-layers) without the training diverging—a feat previous methods could not achieve.
Key Insights
- Updates, Not Just Gradients: The root cause of instability in deep models isn't just exploding gradients, but exploding model updates relative to the weight scales. If the update is too large at the beginning, the model becomes "trapped" in a bad state.
- Depth is More Efficient than Width: One of the most striking findings is that a "deep and narrow" model (e.g., 200 layers, 3.2B parameters) can significantly outperform a "shallow and wide" model (e.g., 48 layers, 12B parameters). This suggests that increasing depth is a more parameter-efficient way to gain model capacity.
- Logarithmic Scaling Law: The paper identifies a scaling law where the BLEU score for multilingual translation grows logarithmically with the depth of the model.
Performance Benchmarks
- SOTA Performance: On the WMT-17 En-De task, DeepNet (100L-100L) achieved 28.9 BLEU, while standard Post-LN models diverged and Pre-LN models lagged behind.
- Efficiency Gains: In the massively multilingual setting, a 200-layer DeepNet with 3.2B parameters outperformed the state-of-the-art 12B parameter M2M-100 model by 5 BLEU points.
Conclusion and Future Work
DeepNet demonstrates that depth is a viable and efficient scaling dimension for Transformers. While the current work focuses on Machine Translation, the stability provided by DeepNorm opens doors for scaling in other domains such as language model pre-training (BERT/GPT) and Vision Transformers (ViT). The team plans to extend DeepNet to diverse tasks including protein structure prediction and multi-modal pre-training.
Summary updated