Modern deep learning relies heavily on overparameterization. We routinely train networks with far more parameters than data points, yet these models generalize remarkably well instead of overfitting to noise. Why does this happen?
A powerful explanation has emerged from recent theoretical insights: gradient-based optimization imposes a strong, implicit geometric constraint on how networks learn. Instead of scattering randomly across the high-dimensional parameter space, the weight vectors of hidden neurons naturally align themselves along a small number of discrete, highly informative directional rays. This phenomenon—often termed directional weight condensation—effectively collapses the network's functional complexity, forcing an overparameterized model to behave like a low-capacity, highly regularized system.
As demonstrated by Zhongwang Zhang in Loss Spike in Training Neural Networks, tracking this condensation process provides a direct window into how a network's internal complexity simplifies over the course of training.
The Mechanics of Early Alignment
Directional condensation is not a gradual byproduct of long training cycles; it typically occurs during the very initial phases of optimization, especially when a network is initialized with small weights.
In Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks, Etienne Boursier et al. show that early in training, neurons primarily adjust their directional vectors to lock onto specific extremal directions determined by the data distribution. This directional concentration happens long before the magnitudes of the weights experience significant growth. By clumping the vast majority of neurons into a handful of shared directions early on, the network drastically reduces its effective width, preventing it from using its massive capacity to memorize high-frequency noise.
This early alignment is a fundamental characteristic of gradient dynamics across architectures, including modern sequence models. In From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics, Zheng-An Chen et al. show that the rows or columns of a transformer's weight matrices undergo a similar directional alignment toward target orientations dictated by the data. This early alignment serves as an implicit regularizer that curtails representational complexity before magnitude growth takes over.
Interestingly, the geometry of this condensation is strictly bounded by the network's activation function. Hanxu Zhou et al. proved in Towards Understanding the Condensation of Neural Networks at Initial Training that when starting from small initial weights, the maximum number of stable directions that hidden neurons can condense into is exactly twice the multiplicity (the order of the first non-zero derivative at the origin) of the activation function.
Collapsing Effective Width via Structural Bundling
When hundreds of hidden neurons align their input weights along the same directional ray, they cease to function as independent features. Instead, they form a collective bundle that acts as a single, unified component.
Arthur Jacot et al. formalized this in Feature Learning in $L_2$-regularized DNNs: Attraction/Repulsion and Sparsity, demonstrating that $L_2$ regularization creates a balancing act of attraction and repulsion. This forces neurons to align along specific rays, allowing groups of redundant units to be represented by a single effective neuron without any loss of information. This structural collapse explains why empirical performance often plateaus at widths far narrower than worst-case theoretical bounds.
Several standard training techniques and data conditions implicitly accelerate this structural bundling:
- Dropout Regularization: Rather than simply preventing co-adaptation through random noise, dropout actively shapes the parameter geometry. In Implicit regularization of dropout, Zhongwang Zhang et al. show that stochastic perturbations from dropout force neuron weight vectors to align into a small number of discrete orientations, reducing functional capacity.
- Data Separability: In The inductive bias of ReLU networks on orthogonally separable data, Mary Phuong et al. show that overparameterized networks undergo a structural collapse where neurons converge to identical weight directions to isolate class-specific max-margin directions.
- Correlated Inputs: Even when dealing with highly correlated inputs, Dmitry Chistikov et al. demonstrate in Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs that overparameterized networks exhibit a bundling effect, collectively aligning toward a common target direction to simplify functional complexity.
This clustering fundamentally simplifies the network's decision boundaries. As Alon Brutzkus et al. show in Towards Understanding Learning in Neural Networks with Linear Teachers, when neurons with the same output sign cluster together, their aggregate response forces the highly expressive network to adopt a simple, approximately linear decision boundary that generalizes robustly.
Bypassing the Curse of Dimensionality
Why is directional alignment so beneficial for generalization? Real-world data often arrives in high-dimensional ambient spaces, but the underlying target function typically depends on a low-dimensional "relevant subspace." Directional alignment acts as an adaptive dimensionality reduction mechanism that discovers this subspace.
Alex Damian et al. proved in Neural Networks can Learn Representations with Gradient Descent that during the initial stages of training, gradient descent leverages second-order derivatives to align hidden layer weights with the principal subspace of a target function. This alignment transforms a high-dimensional optimization problem into a lower-dimensional task focused strictly on relevant features.
This active feature alignment is precisely why neural networks fundamentally outperform classical kernel methods on complex data. As Maria Refinetti et al. demonstrate in Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed, gradient descent drives hidden neuron weights to align directly with the mean vectors of data distributions, collapsing the high-dimensional space and drastically reducing sample complexity. Similarly, Jimmy Ba et al. in Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective show that this gradient-driven alignment allows networks to extract low-dimensional signals under much weaker signal strengths than kernel methods require.
By combining optimization steps with regularization, networks naturally filter out ambient noise. Alireza Mousavi Hosseini et al. show in Neural Networks Efficiently Learn Low-Dimensional Representations with SGD that Stochastic Gradient Descent (SGD) combined with weight decay forces weights to align within the target function's relevant low-dimensional subspace, bypassing the curse of dimensionality.
Providing a grand unified view of this dynamic, Luca Arnaboldi et al. in From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks demonstrate that throughout high-dimensional training, weights do not wander randomly but remain concentrated within a low-dimensional subspace defined by the principal directions of the target function.
The Simplicity Bias of the Condensed Regime
Ultimately, directional weight condensation provides a concrete, structural explanation for the "simplicity bias" observed in deep networks—their innate preference for simple functions over complex ones.
In Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data, Nikita Tsoy et al. prove that hidden neurons in wide networks naturally gravitate toward the global extrema of a data-dependent function. As a result, they cluster along a small number of dominant feature directions regardless of the network's actual width, restricting the model's functional capacity to a few prominent features.
As Hanxu Zhou et al. conclude in Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width, this condensed regime serves as a structural indicator that the network is successfully learning low-complexity representations and structured features from the input data, rather than remaining in a lazy, linear state. By aligning their weights, overparameterized networks naturally prune their own expressive capacity, unlocking the efficient generalization that makes deep learning work.
Key Foundational Work
Implicit Regularization of Dropout (2022) — contributed 2 concept elements to this direction.
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width (2022) — contributed 2 concept elements to this direction.
Towards Understanding the Condensation of Neural Networks at Initial Training (2022) — contributed 2 concept elements to this direction.