A Mean Field Theory of Batch Normalization

Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

Abstract

We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

Introduction

Batch Normalization (BN) is a cornerstone of modern deep learning, widely credited with enabling the training of very deep architectures by stabilizing internal activations and allowing for higher learning rates. Despite its empirical success, the theoretical mechanisms underlying BN have remained contentious. Early hypotheses suggested that BN mitigates "internal covariate shift," while subsequent research pointed toward a smoothing effect on the loss landscape.

However, a critical gap exists in understanding how signals propagate through many successive BN layers. Unlike standard layers, BN is a non-local operation; the transformation of a single input depends on the entire batch. This paper provides a rigorous characterization of signal propagation and gradient backpropagation in wide, batch-normalized networks at initialization, revealing a surprising "dark side": BN inherently causes gradients to explode exponentially with depth.

Methodology

To analyze the behavior of BN in deep networks, the authors utilize a Mean Field Theory framework, examining networks in the "infinite width" limit where pre-activations are treated as jointly Gaussian. The non-local nature of BN necessitates a sophisticated mathematical toolkit:

  • Gegenbauer Polynomials: These are used to handle the geometry of BN, which involves projecting data onto a high-dimensional sphere. Gegenbauer transforms allow the decomposition of the BN operation into orthogonal components.
  • Ultrasymmetric Operators: The authors define this new class of linear operators to describe the symmetries inherent in processing a batch of data.
  • Laplace and Fourier Transforms: These transforms are employed to derive closed-form solutions for the propagation of signals through common activation functions like ReLU.

The Gradient Explosion Phenomenon

The core finding of this research is that gradient signals grow exponentially with depth in batch-normalized networks. In standard networks, gradient vanishing or explosion can often be mitigated by tuning the initial weight variance ($\sigma_w^2$). However, because BN is scale-invariant, it is indifferent to weight scales. The explosion is a fundamental property of the BN transformation itself.

For a ReLU-BN network, the theory predicts that the gradient norm grows at a rate of approximately $1.467^L$, where $L$ is the depth. Even in a perfectly linear network, the growth rate is $(B-2)/(B-3)$ (where $B$ is batch size), which is strictly greater than 1. This leads to several key observations:

  • Exponential Information Loss: As signals propagate deeper, different samples in a batch become increasingly decorrelated. This "chaos" causes the network to lose the relationship between inputs, eventually making learning impossible.
  • The "Linear" Escape Hatch: The rate of explosion is minimized when the activation function is linear. By tuning hyperparameters to operate in a "near-linear" regime—such as using a very small gain $\gamma$ or a very large bias $\beta$—the maximum trainable depth can be extended.
  • Disproving Previous Conjectures: The authors explicitly disprove the conjecture from the original BN paper that BN leads to layer Jacobians with singular values close to 1. In fact, BN forces these values away from 1, contributing to the explosion.
  • Dynamical Equilibrium: Interestingly, while gradients explode at initialization (step 0), the network's weights grow exponentially after just one step of optimization. This reaches a stable equilibrium where gradients become better behaved, explaining why BN networks can often be trained despite the initial explosion, provided numerical overflow does not occur.

Empirical Validation and Trainability

The theoretical predictions were verified through simulations on MNIST and CIFAR-10 datasets. The theory accurately predicts the "trainable depth limit"—the point at which a network becomes untrainable due to gradient instability. For vanilla ReLU-BN networks on MNIST, this limit is approximately 50 layers.

Furthermore, the authors investigated the learning dynamics of VGG19, observing the predicted transition to a stable equilibrium after the first optimization step. The study also highlights the role of $\epsilon$ (the constant added for numerical stability), finding that increasing $\epsilon$ can actually serve as a hyperparameter to dampen gradient explosion.

Conclusion

This work challenges the conventional wisdom that BN solely "stabilizes" training. By proving that BN causes gradient explosion in deep, fully-connected networks, the authors provide a precise limit on the depth of vanilla BN architectures. While these effects can be mitigated by operating near the linear regime or relying on the rapid transition to dynamical equilibrium, the findings underscore why residual connections (ResNets) remain essential for training extremely deep models. Future work is needed to extend this mean field analysis to convolutional architectures and to further explore the interaction between BN and skip connections.

Summary updated

Abstract

We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.
BibTeX
@inproceedings{yang2019mean,
  author = {Greg Yang and Jeffrey Pennington and Vinay Rao and Jascha Sohl-Dickstein and Samuel S. Schoenholz},
  title = {{A Mean Field Theory of Batch Normalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=3-tYJSI8z4A},
}
Figure