On Layer Normalization in the Transformer Architecture

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu

Abstract

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.

Introduction

Training a Transformer from scratch typically requires a meticulously tuned learning rate warm-up stage. Without starting at a very low learning rate and gradually increasing it over thousands of steps, the model often fails to converge. This requirement complicates the training process, introduces additional hyperparameters, and slows down overall optimization. While empirical evidence suggested that a variant called "Pre-LN" was easier to train, a rigorous theoretical explanation for this phenomenon was lacking. This work bridges that gap by analyzing the gradient scales at initialization using mean field theory.

Architectural Comparison: Post-LN vs. Pre-LN

The core difference between the standard Transformer and the more stable variant lies in the placement of the Layer Normalization (LN) relative to the residual connections.

  1. Post-LN (Original): Layer Normalization is applied after the residual addition: $x_{l+1} = \text{LayerNorm}(x_l + \text{Sublayer}(x_l))$.
  2. Pre-LN (Variant): Layer Normalization is applied before the sub-layer within the residual branch: $x_{l+1} = x_l + \text{Sublayer}(\text{LayerNorm}(x_l))$.
Comparison of (a) Post-LN and (b) Pre-LN Transformer architectures.

Theoretical Analysis: The Gradient "Hot Spot"

Using Mean Field Theory, the authors analyzed the expected norm of gradients at initialization. They discovered that in the Post-LN architecture, gradients near the output layer (the "final" layers) are dangerously large. This "hot spot" occurs because the input to the final LayerNorm is not well-scaled at the start of training. Applying a standard learning rate to these massive gradients causes weights to jump into unstable regions, leading to divergence.

Conversely, in the Pre-LN architecture, the scale of hidden states grows as the model deepens. This growth actually stabilizes the final LayerNorm, effectively scaling down the gradients of the last layer by a factor of approximately $\sqrt{L}$, where $L$ is the number of layers. Consequently, deeper Pre-LN models are inherently more stable at initialization.

Gradient expectation across layers at initialization. Post-LN shows a significant "hot spot" in later layers, while Pre-LN remains stable.

Experimental Results

The theoretical findings were validated across several benchmarks, including IWSLT14 (De-En), WMT14 (En-De), and BERT pre-training.

  • Eliminating Warm-up: Pre-LN models were successfully trained without any warm-up stage, achieving BLEU scores comparable to Post-LN models that required 4,000 warm-up steps.
  • Training Efficiency: In BERT pre-training, the Pre-LN architecture reached the target validation loss in 500k updates, whereas the Post-LN baseline required 700k updates—a 40% speedup.
  • Robustness: Pre-LN demonstrated significantly less sensitivity to the maximum learning rate, reducing the need for extensive hyperparameter sweeps.

Conclusion

The learning rate warm-up stage in traditional Transformers acts as a temporary fix for an underlying architectural instability. By repositioning the Layer Normalization (Pre-LN), the gradients at initialization become well-behaved, allowing for faster, more robust training without the need for a warm-up phase. This shift simplifies the training pipeline and offers significant computational savings.

Summary updated

Abstract

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.
BibTeX
@inproceedings{xiong2020layer,
  author = {Ruibin Xiong and Yunchang Yang and Di He and Kai Zheng and Shuxin Zheng and Chen Xing and Huishuai Zhang and Yanyan Lan and Liwei Wang and Tie-Yan Liu},
  title = {{On Layer Normalization in the Transformer Architecture}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=FCvs-jrH6f_},
}
Figure