PowerNorm: Rethinking Batch Normalization in Transformers

Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, Kurt KeutzerICML 2020 · OpenReview

TL;DR

While Batch Normalization (BN) is the gold standard for Computer Vision (CV), it notoriously degrades performance in NLP Transformers. This paper identifies the culprit—wildly fluctuating batch statistics in NLP data—and introduces Power Normalization (PN), a novel scheme that uses running quadratic means and approximate backpropagation to significantly outperform Layer Normalization (LN).

The Normalization Divide: CV vs. NLP

In the world of deep learning, normalization is the "secret sauce" that makes training stable and fast. However, there is a weird schism: Computer Vision models almost exclusively use Batch Normalization (BN), while Natural Language Processing (NLP) models rely on Layer Normalization (LN).

Comparison of normalization dimensions

When researchers try to use "vanilla" BN in Transformers, performance tanks. Until now, the community lacked a thorough understanding of why this happens. The authors of this paper set out to diagnose BN's failure in NLP and propose a superior alternative that bridges the gap between the two fields.

Diagnosing the Problem and the PowerNorm Solution

The authors first performed a systematic study and found that NLP data exhibits massive fluctuations in batch statistics (mean and variance) throughout training, unlike CV data which stabilizes quickly. To solve this, they developed Power Normalization (PN) through four key modifications:

  1. Relaxing Zero-Mean: Instead of forcing the data to have a mean of zero (which is highly unstable in NLP), PN focuses on the "power" of the signal.
  2. Quadratic Mean (PN-V): They replace the standard variance calculation with a quadratic mean (root mean square). They found that the unit quadratic mean is significantly more stable for NLP signals.
  3. Running Statistics in Forward Pass: Unlike BN, which uses the current batch's statistics during training, PN uses a running moving average of the quadratic mean during the forward pass.
  4. Approximate Backpropagation: Using running statistics in the forward pass usually makes backpropagation mathematically "messy" and computationally expensive. The authors propose an approximate backprop scheme that ensures gradients remain bounded and the model converges efficiently.

Key Insights and Ideas

The "Outlier" Problem: The "aha moment" came from visualizing the Euclidean distance between batch statistics and running statistics.

Fluctuations in batch statistics for CV vs. NLP

In Transformers (like those trained on the IWSLT14 dataset), these distances are orders of magnitude larger than in ResNets (trained on CIFAR10), with extreme outliers that persist throughout the entire training process. This instability is what breaks standard Batch Normalization.

Smoothing the Landscape: Theoretically, the authors prove that PN leads to a smaller Lipschitz constant for the loss function. In plain English: it makes the "hills and valleys" of the optimization landscape smoother, allowing for more stable training.

Better Embeddings: Using Singular Value Decomposition (SVD), the authors showed that PN leads to a better-conditioned embedding layer. The singular values decay more slowly compared to LN, meaning the model is using its representational space more effectively rather than collapsing into a small subspace.

Robustness to Batch Size: One of BN's biggest weaknesses is that it fails when batch sizes are small. PN is remarkably robust, maintaining high performance even when the mini-batch size is reduced.

Key Results

The researchers evaluated PowerNorm across several standard NLP benchmarks, including Machine Translation (IWSLT14, WMT14) and Language Modeling (PTB, WikiText-103). PowerNorm doesn't just match LayerNorm; it consistently beats it across the board without requiring changes to other hyperparameters:

  • Machine Translation: PN improved BLEU scores by 0.4 on IWSLT14 and 0.6 on WMT14 compared to the standard LayerNorm.
  • Language Modeling: The gains here were even more dramatic. PN reduced Perplexity (PPL) by 5.6 points on PTB and 3.0 points on WikiText-103.
  • Comparison to BN: On WMT14, vanilla BN achieved a BLEU score of 28.1, while PN reached 30.1—a massive jump in translation quality.

Limitations and Future Work

While PowerNorm shows clear advantages, the authors note a few areas for further exploration:

  • Implementation Complexity: PN requires a specific approximate backpropagation formula which is slightly more complex to implement than standard LN or BN.
  • Scope: The study focuses heavily on Transformers. While Transformers are the dominant architecture in NLP, testing PN on other sequence models would be a logical next step.
  • Hyperparameter Sensitivity: Although PN works well with LN's default settings, the moving average coefficient for the running statistics is a new parameter that might require tuning for specific, niche datasets.

Summary updated

Abstract

The standard normalization method for neural network (NN) models used in Natural Language Processing (NLP) is layer normalization (LN).This is different than batch normalization (BN), which is wide...
BibTeX
@inproceedings{shen2020powernorm,
  author = {Sheng Shen and Zhewei Yao and Amir Gholami and Michael W. Mahoney and Kurt Keutzer},
  title = {{PowerNorm: Rethinking Batch Normalization in Transformers}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=5WIM18Q6R8m},
}
Figure