On the impact of activation and normalization in obtaining isometric embeddings at initialization

Amir Joudaki, Hadi Daneshmand, Francis Bach

Abstract

Training deep neural networks is often hindered by the phenomenon of "rank collapse," where internal representations of distinct inputs become increasingly similar as they pass through successive layers. This paper establishes a theoretical foundation for why Layer Normalization (LN) and non-linear activation functions are effective at mitigating this issue. By introducing a formal metric for isometry and utilizing Hermite expansion, the authors demonstrate that these components work synergistically to push a network's Gram matrix toward the identity matrix at an exponential rate, ensuring that embeddings remain well-spread and distinct even at great depths.

Introduction: The Challenge of Rank Collapse

A persistent challenge in deep learning is maintaining the discriminative power of representations across many layers. As signals propagate through a network, the representations of different inputs often converge, leading to a degenerate or "rank-deficient" Gram matrix. When this occurs, the model loses the ability to distinguish between different inputs, causing training to stall or fail entirely. While empirical evidence has long suggested that normalization layers like Batch Norm and Layer Norm (LN) alleviate this, the specific theoretical mechanism—particularly regarding the role of non-linear activations in conjunction with LN—has remained under-explored. This research quantifies how these architectural choices bias networks toward "isometric" embeddings at initialization.

Quantifying Isometry

To analyze this phenomenon, the authors define a formal metric for Isometry, denoted as $I(G)$. This metric is calculated as the ratio of the normalized determinant of the Gram matrix to its normalized trace.

Mathematically, $I(G)$ measures how effectively the representations fill the available high-dimensional volume. A value of $I(G) = 1$ indicates that representations are perfectly orthogonal (resembling an identity matrix), while a value approaching $0$ signifies that the representations have collapsed into a lower-dimensional subspace. This metric allows for a rigorous tracking of representation health throughout the network's depth.

Theoretical Framework: Isometry Strength and Hermite Expansion

The paper employs Hermite expansion to decompose non-linear activation functions into a sum of polynomials. This decomposition allows the authors to derive a new property termed Isometry Strength ($\beta_\sigma$).

Using mean-field theory—which analyzes networks in the limit of infinite width—the authors apply a Lyapunov function to track the evolution of the Gram matrix. They prove that the "isometry gap" (the distance between the current Gram matrix and the ideal identity matrix) shrinks exponentially as depth increases. This theoretical result suggests that the deeper the network, the more strongly it is biased toward isometric representations, provided the activation function satisfies certain criteria (specifically $\beta_\sigma > 1$).

The Role of Layer Normalization and Non-linearity

Layer Norm as an Isometry Engine

The research formalizes the "isometry bias" of Layer Normalization in Theorem 1 and Corollary 2. These findings show that the act of projecting representations onto a unit sphere—a core component of LN—mathematically guarantees an increase in isometry if there is any initial variance in the input norms. Crucially, this bias is not merely a transient effect at initialization but persists throughout the training process.

Non-linearity as a Volume-Filling Feature

Contrary to the intuition that non-linearities might distort and collapse data, the authors find that they are essential for maintaining isometry. Non-linear activations like ReLU or GeLU help "spread out" data points in high-dimensional space. The paper demonstrates that activations with higher Isometry Strength values correlate directly with faster convergence during Stochastic Gradient Descent (SGD) in shallow architectures.

Empirical Validation

The authors validated their theoretical predictions across several benchmarks:

  • CIFAR-10 and MNIST: Used to track isometry in Multi-Layer Perceptrons (MLPs) and verify the correlation between $\beta_\sigma$ and training speed.
  • Wiki (Simple English): The theory was tested on a pre-trained GPT-2 model. The results confirmed that LayerNorm layers in the final stages of the Transformer architecture significantly decrease the isometry gap, validating the theory in complex, real-world settings. The specific ordering of normalization and activation in Transformers was found to be theoretically optimal for preventing degeneracy.

Conclusion and Future Directions

This work provides a rigorous explanation for the success of Layer Normalization and non-linear activations in modern architectures. However, the authors note that in extremely deep networks, other bottlenecks like gradient explosion may eventually overshadow the benefits of isometry. Furthermore, the current theory focuses on the centering and scaling aspects of LN, leaving the impact of learnable "gain" and "bias" parameters for future study. A promising avenue for subsequent research is the development of "self-normalizing" activations that could achieve isometry without the computational overhead of explicit normalization layers, potentially using the Hermite expansion framework as a design guide.

Summary updated

Abstract

In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.
BibTeX
@inproceedings{joudaki2023impact,
  author = {Amir Joudaki and Hadi Daneshmand and Francis Bach},
  title = {{On the impact of activation and normalization in obtaining isometric embeddings at initialization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=rY4sA9qYKy},
}
Figure