On Layer Normalization in the Transformer Architecture

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu

TL;DR

This paper identifies why standard Transformers require a fragile "learning rate warm-up" stage: the original architecture (Post-LN) creates dangerously large gradients near the output layer at initialization. By moving Layer Normalization inside the residual blocks (Pre-LN), the authors prove that gradients become well-behaved. This simple change allows researchers to scrap the warm-up stage entirely, simplify hyperparameter tuning, and speed up training by up to 40%.

What Problem Does This Paper Address?

Training a Transformer from scratch is notoriously finicky. Unlike standard convolutional networks, Transformers usually require a "warm-up" period where the learning rate starts near zero and gradually increases over thousands of iterations. Without this careful initialization phase, the model often fails to converge.

This warm-up stage is problematic for two main reasons:

  1. Hyperparameter Sensitivity: The final performance of the model is highly sensitive to the maximum learning rate and the exact duration of the warm-up, requiring costly and time-consuming tuning.
  2. Efficiency: It artificially slows down the early stages of optimization, making an already computationally expensive training process even longer.

The authors set out to understand why this warm-up is necessary from a mathematical standpoint and whether structural changes to the architecture can eliminate the need for it.

Methods

The researchers employ Mean Field Theory to mathematically analyze the behavior of gradients at the exact moment of model initialization. They compare two primary architectural variants of the Transformer:

  • Post-LN (The Original): Layer Normalization is placed between the residual blocks (after the addition step).
  • Pre-LN: Layer Normalization is placed inside the residual blocks (before the Multi-Head Attention or Feed-Forward sub-layers).
Comparison of Post-LN (a) and Pre-LN (b) architectures

By deriving the expected scale of gradients for both architectures, they provide a theoretical foundation for why one is inherently more stable than the other. They then validate these theories through extensive empirical experiments in machine translation and unsupervised pre-training.

Datasets

The authors evaluated their findings on several standard NLP benchmarks to ensure broad applicability:

  • IWSLT14 German-to-English (De-En): A common small-scale machine translation task.
  • WMT14 English-to-German (En-De): A large-scale machine translation benchmark.
  • BERT Pre-training: Using the English Wikipedia corpus and BookCorpus (approx. 3.4B words) to test large-scale unsupervised learning.
  • Downstream Tasks: Evaluation on MRPC and RTE datasets to ensure the pre-trained Pre-LN BERT models maintained high quality during fine-tuning.

Key Insights and Ideas

  • Post-LN Gradient Explosion at Init: The mathematical analysis reveals that in Post-LN Transformers, the expected gradient of parameters near the output layer is independent of the total number of layers ($L$). Because the input to these layers is not well-scaled at initialization, it leads to large, unstable updates if a high learning rate is used immediately.
  • Pre-LN Stability: In the Pre-LN version, the gradient scale is naturally suppressed by a factor of $\sqrt{L}$. This means as the network gets deeper, the gradients stay within a "well-behaved" range, preventing catastrophic early updates.
  • Warm-up as a "Patch": The learning rate warm-up is essentially a manual fix for the Post-LN architecture's inherent instability. It keeps updates artificially small until the model moves away from its chaotic initial state.
  • Architecture Trumps Optimization: Instead of trying to fix the optimizer (e.g., using complex variants like RAdam), simply changing the location of the Layer Normalization solves the root cause of the instability.

Key Results

  • Removing Warm-up: Pre-LN Transformers were successfully trained without any warm-up stage, achieving BLEU scores and validation losses comparable to Post-LN models that required 4,000 warm-up steps.
  • Faster Convergence: In BERT pre-training, the Pre-LN model reached the same validation loss as the Post-LN baseline significantly faster. Specifically, the authors observed a 40% speed-up in training time for BERT.
  • Robustness: Pre-LN models proved much less sensitive to the choice of maximum learning rate, drastically easing the burden of hyperparameter optimization for practitioners.
Pre-LN training stability without warm-up

Limitations and Future Work

While the paper provides a strong theoretical and empirical case for Pre-LN, a few areas remain for further exploration:

  • Performance Ceiling: Some researchers in the community have suggested that Post-LN might eventually reach a slightly higher performance peak if tuned perfectly over a very long time, though this paper shows Pre-LN reaching comparable results much more easily and reliably.
  • Initialization Schemes: The study focuses primarily on standard Xavier initialization. Future work could investigate if alternative initialization strategies (like Fixup) interact differently with LN placement.
  • Other Architectures: The theoretical analysis is specific to the Transformer's residual structure. Extending this "gradient scale" analysis to other emerging architectures remains an open and exciting direction.

Summary updated

Abstract

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial...
BibTeX
@inproceedings{xiong2020layer,
  author = {Ruibin Xiong and Yunchang Yang and Di He and Kai Zheng and Shuxin Zheng and Chen Xing and Huishuai Zhang and Yanyan Lan and Liwei Wang and Tie-Yan Liu},
  title = {{On Layer Normalization in the Transformer Architecture}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=FCvs-jrH6f_},
}
Figure