In an ideal deep network, each successive layer should refine the representation of data, extracting increasingly abstract features while maintaining the nuances that distinguish one input from another. However, as networks grow deeper, they often fall victim to rank collapse: a geometric phenomenon where hidden representations lose their diversity and begin to align along a single direction.
When rank collapse occurs, the "volume" of the data in the representation space shrinks. Distinct inputs—whether they are different images in a batch or different tokens in a sentence—become indistinguishable. This effectively neutralizes the expressive power of the model, making it impossible for the network to learn complex patterns regardless of its depth.
The Geometry of Information Loss
To understand why rank collapse happens, we can view the evolution of data through a network as a dynamical system. In vanilla deep networks, the repeated application of linear transformations and non-linear activations tends to squash the data. Hadi Daneshmand, Francis Bach, and Thomas Hofmann demonstrated in Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks that without intervention, activation matrices quickly converge to rank 1.
In the context of Transformers, this problem is often framed as over-smoothing. Borjan Geshkovski and colleagues in A mathematical perspective on Transformers model self-attention as a system of interacting particles on a sphere. They show that the attention mechanism acts as an attractive force, pulling token representations toward each other until they cluster into a single point. This clustering destroys the unique information necessary for tasks like translation or reasoning.
Normalization as an Orthogonalizing Force
The most successful defense against rank collapse is normalization. While Batch Normalization (BN) was originally motivated by "internal covariate shift," recent theoretical work suggests its true power lies in its geometric impact on the representation space.
Amir Joudaki, Hadi Daneshmand, and Francis Bach have shown that BN effectively orthogonalizes representations. In Batch Normalization Orthogonalizes Representations in Deep Random Networks, they argue that BN pushes the hidden states of different samples to be as uncorrelated as possible. By forcing the covariance matrix of the activations to stay close to the identity matrix, BN ensures that the network maintains a high-rank representation even at great depths.
This principle extends to Layer Normalization. In On the impact of activation and normalization in obtaining isometric embeddings at initialization, Joudaki and Bach prove that the combination of normalization and specific non-linear activations pushes the network toward isometry—a state where the distances between inputs are preserved as they pass through the layers.
The Role of Initialization and Noise
If the deterministic path of a deep network leads to collapse, how do we stay on the edge of expressivity? The answer often lies in the "chaos" of initialization.
- Orthogonal Initialization: Alexandru Meterez and Hadi Daneshmand found that initializing a network with orthogonal weights, when combined with BN, prevents both rank collapse and the "gradient explosion" that typically plagues deep models. This synergy allows for the training of networks with virtually infinite depth, as detailed in Towards Training Without Depth Limits.
- Stochasticity: Interestingly, the randomness of initialization itself acts as a safeguard. Lev Fedorov and Michael Eli Sander proved in Clustering in Deep Stochastic Transformers that intrinsic noise prevents tokens from collapsing into a single cluster. Instead, tokens can form stable "antipodal" configurations, maintaining signal diversity through the layers.
Detecting and Exploiting Collapse
Rank collapse isn't just a theoretical hurdle; it is a diagnostic tool. Because a collapsing network is a failing network, the degree of rank loss can serve as a proxy for model quality.
In TF-MAS: Training-free Mamba2 Architecture Search, Yi Fan and Yu-Bin Yang use rank collapse as a metric to identify high-performing architectures without ever having to train them. Similarly, in Vision Transformers, Dong-Jae Lee and colleagues developed a "frequency-aware" token reduction strategy in Frequency-Aware Token Reduction for Efficient Vision Transformer. By identifying and preserving "high-frequency" tokens—those that resist the pull toward the mean—they mitigate rank collapse while reducing computational cost.
Even in Mixture-of-Experts (MoE) models, the problem persists as expert collapse, where routers become highly correlated and fail to utilize the full breadth of the model. Giang Do and Hung Le address this in Eigenvectors of Experts are Training-free Non-collapsing Routers by using the eigenvectors of expert weights to ensure diverse, non-collapsing routing.
Summary
Rank collapse is the geometric manifestation of information loss in deep learning. Whether it is tokens clustering in a Transformer or hidden states aligning in an MLP, the result is the same: a loss of expressivity. By using normalization to orthogonalize representations and leveraging orthogonal initializations to maintain isometry, researchers are finding ways to push the limits of depth without sacrificing the diversity of the data.