Imagine a high-dimensional sheet of paper representing a complex dataset. As this data passes through the successive layers of a deep neural network, each layer acts like a pair of hands folding, stretching, and transforming the paper to isolate task-relevant patterns. Ideally, a deeper network provides more opportunities to create intricate, highly non-linear folds.
In practice, however, an opposing force often takes over: representation collapse (or feature collapse). Instead of elegantly folding the paper, the deeper layers of the network crush it into a tight, uniform ball. Distinct data points become indistinguishable, adjacent layers begin to output nearly identical features, and the network's expressive capacity plummets.
This phenomenon manifests across diverse architectures under various names, including inter-layer over-smoothing, kernel flattening, and basis collapse. To build deeper, more capable models, researchers have developed a multi-faceted toolkit designed to maintain feature diversity and prevent representations from collapsing into redundancy.
The Geometry of Collapse
Why do deep networks naturally gravitate toward redundancy? The root cause lies in how information propagates through unconstrained parametric spaces. As layers stack, the transformations applied to hidden states can contract the data distribution.
In Globally Gated Deep Linear Networks, Qianyi Li et al. formalize this as kernel flattening, where the internal representations of entirely distinct inputs become increasingly correlated and linearly dependent as they travel deeper. This loss of discriminative power means that increasing depth yields diminishing returns; the network enters a "lazy training" regime where intermediate neurons behave identically.
This degradation is not limited to standard feedforward networks. In multi-view clustering, Xuelong Li et al. observe in Towards Learnable Anchor for Deep Multi-View Clustering that unconstrained deep representations frequently cause all data points to map to identical coordinates, destroying the clusters entirely. Similarly, in sparse Mixture of Experts (SMoE) models, Giang Do et al. show in SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse that different experts often collapse into computing identical functions, neutralizing the benefits of a sparse, specialized architecture.
Even optimization objectives can inadvertently encourage collapse. As detailed by Jieting Wang et al. in Stabilizing Sample Similarity in Representation via Mitigating Random Consistency, networks suffer from a statistical bias called "random consistency." Models can minimize standard loss functions by defaulting to non-informative states—such as uniform similarity matrices—purely by chance, rather than learning robust, class-distinctive features.
Safeguarding Geometry at Initialization
One of the most effective ways to prevent representation collapse is to ensure that the network preserves the geometry of the input space from the very first forward pass.
[Input Space] ---> [Layer 1: Hypercube Preserved] ---> [Layer N: Full-Rank Features]
| |
X --(Without Initialization Safeguards)----------------> X [Collapsed Line/Point]
In Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint, Eric C Cyr et al. analyze deep neural networks as an adaptive basis followed by a linear solver. To prevent basis collapse—where deep layers map distinct inputs to linearly dependent or constant outputs—the authors introduce a "Box initialization." This method constrains weights and biases so that each layer maps the input unit hypercube directly into itself. By explicitly preserving the geometric boundaries of the data across layers, the network maintains its capacity to represent complex features before training even begins.
A parallel approach is developed by Cong Fang et al. in Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks. They utilize a specialized initialization strategy featuring a regression step to guarantee that initial neuron outputs are diverse and non-redundant. Ensuring a full-rank feature matrix at the start prevents intermediate layers from falling into identical neuron behavior, allowing over-parameterized networks to escape lazy training and converge to global minima.
Architectural and Scaling Solutions
While proper initialization protects the network at the start, maintaining feature diversity during training requires structural mechanisms that prevent gradients and representations from degrading as depth scales.
Scale-Distribution Decoupling
Standard normalization layers often struggle to balance weight magnitudes and directional updates, leading to feature homogenization. In Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models, Ya Wang et al. propose explicitly separating the scale of activations from their distributional structure. By isolating the scaling factor, the model is structurally incentivized to develop more distinct and diverse representations throughout its depth, significantly reducing inter-layer similarity.
Balancing Normalization and Connections
The arrangement of normalization layers dictates how representations evolve across depth. Pengxiang Li et al. tackle the "lazy deep layer" problem in large language models in Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN. By strategically combining Post-Layer Normalization (Post-LN) in early layers with Pre-Layer Normalization (Pre-LN) in deeper ones, they maintain healthy gradient norms throughout the network. This prevents the representations of successive layers from developing high cosine similarity, ensuring each layer performs a distinct transformation.
For alternative training paradigms like Geoffrey Hinton's Forward-Forward algorithm, Liang Sun et al. demonstrate in DeeperForward: Enhanced Forward-Forward Training for Deeper and Better Performance that replacing squared goodness metrics with mean goodness combined with layer normalization prevents feature collapse. This mechanism decouples the training signal from the feature representation, forcing subsequent layers to learn novel information rather than recycling previous signals.
Breaking the Residual Bottleneck
While residual connections are designed to alleviate gradient degradation, they can become a communication bottleneck that encourages layers to replicate information. To break this bottleneck, Da Xiao et al. introduce MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections. Their dynamic multiway connections maintain high variance between the hidden states of different layers, ensuring that increasing model depth results in the acquisition of new, distinct features rather than redundant information.
Infinite-Depth Parametrization
To scale networks safely toward infinite depth, Greg Yang et al. introduce Depth-$\mu$P in Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. This framework provides a precise recipe for scaling residual branch multipliers and learning rates. By maximizing both feature learning and the decorrelation of activations between adjacent layers, Depth-$\mu$P prevents successive layers from converging to redundant representations, allowing models to scale seamlessly in depth.
Explicit Regularization and Loss Design
When architectural adjustments alone cannot prevent collapse, researchers turn to explicit optimization constraints that penalize homogeneity and force representations to differentiate.
Inter-Layer Contrastive Learning
To directly combat inter-layer over-smoothing in text embeddings, Nuo Chen et al. propose Self-Contrastive Learning (SSCL) in Alleviating Over-smoothing for Unsupervised Sentence Representation.
[Layer t-2] ---\
+---> Treated as Negative Examples ---\
[Layer t-1] ---/ v
[Maximize Distance]
^
[Layer t] -------------------------------------------/
SSCL uses a model's own intermediate layers as negative examples in a contrastive objective, maximizing the distance between the final layer and intermediate states. This forces deeper layers to continuously transform information rather than merely replicating the inputs of previous stages.
Targeted Layer Constraints and Subspace Boosting
Because representational collapse is often empirically more severe in the earlier stages of a network, Giang Do et al. concentrate diversification efforts on initial layers. In SimSMoE, they apply strong similarity constraints—measured via Centered Kernel Alignment (CKA)—to early layers, preserving feature diversity before it naturally propagates to deeper stages.
In time-series forecasting, representation collapse often manifests as "hallucinations" where hidden states become too homogeneous and drop underlying data dynamics. In Investigating Hallucinations of Time Series Foundation Models through Signal Subspace Analysis, Yufeng Zou et al. mitigate this by intentionally boosting the activations of signal-carrying neurons. This test-time intervention magnifies signal subspaces, ensuring that critical input information is preserved as it moves through the model's layers.
Eliminating Singularities and Random Bias
In convolutional networks, feature collapse can lead to "elimination singularities," where neurons enter a self-reinforcing cycle of zero-weight stagnation. To prevent this, Chengkun Sun et al. design a mechanism in Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities that actively perturbs feature distributions before they reach the activation layer, maintaining healthy gradient flow.
Finally, to counteract the mathematical bias of random consistency, Jieting Wang et al. introduce the Pure Square Euclidean Distance (PSED) loss function in Stabilizing Sample Similarity in Representation via Mitigating Random Consistency. By explicitly penalizing non-informative states like identity or uniform similarity matrices, the PSED objective forces the network to maintain discriminative features and distinct class clusters across its entire depth.
Why It Works: The Denoising and Orthogonality Effect
Ultimately, these diverse strategies—whether applied via initialization, architecture, or loss functions—converge on a shared structural outcome: orthogonality and richness in the weight and feature spaces.
When a network successfully resists representation collapse, its internal machinery undergoes a fundamental shift. As demonstrated by Paul Gavrikov et al. in Adversarial Robustness through the Lens of Convolutional Filters, models that preserve structural richness exhibit a monotonic increase in filter orthogonality as depth increases. Rather than becoming redundant, the filters become more diverse and less sparse.
By preventing layers from collapsing into identical representations, these techniques ensure that each additional layer provides a genuine, distinct transformation of the data. The network is forced to build internal denoising mechanisms and maintain high-dimensional expressivity, unlocking the true mathematical promise of deep architectures.
Key Foundational Work
Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint (2020) — contributed 3 concept elements to this direction.
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks (2021) — contributed 2 concept elements to this direction.