On the impact of activation and normalization in obtaining isometric embeddings at initialization

Amir Joudaki, Hadi Daneshmand, Francis R. Bach

TL;DR

This paper provides a theoretical foundation for why layer normalization and non-linear activations are so effective at the start of training. The authors prove that these components work together to push a neural network's internal representations toward "isometry"—a state where inputs remain distinct and well-spread rather than collapsing into a single point—at an exponential rate as the network gets deeper.

What Problem Does This Paper Address?

Training deep neural networks is notoriously difficult because of "rank collapse." As signals pass through many layers, the representations of different inputs often become increasingly similar. Mathematically, the Gram matrix (which tracks the pairwise similarity between all inputs in a batch) becomes degenerate or "rank-deficient." If every input looks the same to the deeper layers of a network, the model can't learn to distinguish between them, and training crawls to a halt.

While researchers knew that normalization layers (like Batch Norm or Layer Norm) helped mitigate this, existing theory didn't fully explain the role of Layer Normalization (LN)—the backbone of modern Transformers—or how specific non-linear activation functions (like ReLU or GeLU) influenced this process. This paper fills that gap by quantifying how LN and activations bias the network toward healthy, "isometric" embeddings at initialization.

Methods

The authors introduce a formal metric for Isometry, $I(G)$, defined as the ratio of the normalized determinant of the Gram matrix to its normalized trace. Essentially, it measures how "volume-filling" the representations are; a value of 1 means the representations are perfectly orthogonal (like an identity matrix), while 0 means they have collapsed.

To analyze the impact of activations, they use Hermite expansion, a way of decomposing non-linear functions into a sum of polynomials. From this, they derive a new property called Isometry Strength ($\beta_\sigma$). By applying mean-field theory (analyzing the network in the limit of infinite width), they use a Lyapunov function to prove that the isometry gap (the distance from the ideal identity matrix) shrinks exponentially with depth.

Datasets

The authors validated their theoretical findings using several standard benchmarks:

  • CIFAR-10: Used for training Multi-Layer Perceptrons (MLPs) with various activations to track isometry during SGD.
  • MNIST: Used to test the theory on networks with variable widths.
  • Wiki (Simple English): Used to evaluate a pre-trained GPT-2 model, demonstrating that the theory holds even in complex, real-world architectures.

Key Insights and Ideas

  • Layer Norm is an Isometry Engine: The paper proves that LN doesn't just stabilize training; it actively pushes the Gram matrix toward the identity matrix. Crucially, this bias persists not just at initialization, but throughout the training process.
  • Non-linearity is a Feature, Not a Bug: Surprisingly, non-linear activations actually increase isometry. While a linear network might allow representations to collapse, non-linearities (quantified by $\beta_\sigma > 1$) help "spread out" the data points in high-dimensional space.
  • Predicting Training Speed: The authors found a striking correlation between their "Isometry Strength" metric and the speed of Stochastic Gradient Descent (SGD). Activations with higher $\beta_\sigma$ values generally led to faster convergence in shallow networks.
  • The Transformer Connection: The specific ordering of normalization and activation used in Transformers (analyzed in Section 7) is shown to be theoretically optimal for maintaining a non-degenerate Gram matrix.

Key Results

  • Theorem 1 & Corollary 2: These formalize the "isometry bias," showing that projecting representations onto a unit sphere (a key part of LN) mathematically guarantees an increase in isometry if there is any variance in the input norms.
  • Exponential Convergence: Theorem 4 proves that for a wide class of activations, the representations converge to a non-degenerate fixed point at an exponential rate relative to the depth of the network.
  • Empirical Validation: Experiments on GPT-2 (Figure B.5) confirmed that LayerNorm layers in the final stages of the model significantly decrease the "isometry gap," validating the theory in a non-random, pre-trained setting.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Depth Limits: While the correlation between isometry and training speed is strong for shallow networks (up to 10 layers), it breaks down in extremely deep networks where other issues, like gradient explosion, become the dominant bottleneck.
  • Learnable Parameters: The current theory focuses on the centering and scaling aspects of Layer Norm but does not fully account for the learnable "gain" and "bias" parameters ($\gamma$ and $\beta$) that are adjusted during training.
  • Self-Normalizing Activations: The paper suggests a promising future direction: designing "self-normalizing" activations that can achieve isometry without the computational overhead of explicit normalization layers. They point to SeLU as a starting point that could be further refined using their Hermite expansion framework.

Summary updated

Abstract

In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.
BibTeX
@inproceedings{joudaki2023impact,
  author = {Amir Joudaki and Hadi Daneshmand and Francis R. Bach},
  title = {{On the impact of activation and normalization in obtaining isometric embeddings at initialization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=rY4sA9qYKy},
}
Figure