Overparameterized models have an infinite number of ways to fit a training dataset. Yet, gradient descent consistently finds solutions that generalize remarkably well, guided by an implicit regularization inherent to the optimization process. What determines which solution the algorithm selects?
The scale of parameter initialization acts as a primary control mechanism for this selection. When initialized near zero, the optimization path is steered toward sparse, low-rank solutions. Conversely, larger initializations cause the algorithm to behave like standard Euclidean gradient descent, losing this structural bias.
This transition forms a continuous spectrum. As demonstrated in Kernel and Rich Regimes in Overparametrized Models by Blake Woodworth et al., the initialization scale bridges two fundamentally different learning behaviors: a "kernel" (or lazy) regime at large scales, and a "rich" (or feature-learning) regime at infinitesimal scales.
The Dynamics of Small Initialization: A Race for Growth
To understand why a tiny initialization scale yields low-rank or sparse structures, we must look at the early training dynamics. When weights start near zero, the gradient updates for different components are heavily dependent on their current magnitudes.
In Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing, Ziye Ma et al. explain that a small initialization acts as an implicit rank-1 filter. Because gradient descent amplifies dominant signal components significantly faster than noise, the principal directions of the data quickly outpace the rest. The minor components remain dormant or suppressed for extended periods.
This creates what Noam Razin et al. describe in Implicit Regularization in Tensor Factorization as a greedy selection mechanism. Instead of updating all parameters uniformly, the network sequentially activates components based on their alignment with the target signal. Dominik Stöger et al. in Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction show that this initial phase effectively mimics a spectral method, aligning the model's subspace with the target signal's principal components before any local refinement occurs.
This structural simplification has profound effects beyond matrix factorization. For instance, in language modeling, Junjie Yao in An Analysis for Reasoning Bias of Language Models with Small Initialization demonstrates that small initializations bias networks toward discovering low-rank, compositional rules rather than memorizing individual data points.
The Large Initialization Regime: Why Weights Stay Lazy
What happens when we turn the knob the other way? When a model is initialized with large parameter magnitudes, it enters the "lazy learning" or Neural Tangent Kernel (NTK) regime.
In this setting, the initial random weights already provide a massive, high-dimensional basis that spans the entire functional space. Because the weights are large, the network can fit the training data perfectly by making infinitesimally small fractional adjustments to its parameters. The optimization process minimizes the Euclidean distance ($\ell_2$ norm) from the starting configuration, meaning the weights barely move from their initial values.
As a result, the network never undergoes significant representational changes. It simply interpolates the data using its fixed, initial random features. This explains why large initializations favor rote memorization over structural generalization: the model preserves its initial random orthogonality, trapping it in a regime that fits individual data points without compressing them into generalizable, low-rank rules.
Structural Rank as an Inductive Bias
The initialization condition is not solely defined by scalar magnitude; the geometric structure of the initial weight matrix also dictates the learning trajectory.
Yuhan Helena Liu et al. in How connectivity structure shapes rich and lazy learning in neural circuits show that the structural rank of initial weights acts as a powerful inductive bias. High-rank (random) initializations provide a broad functional basis that allows the network to solve tasks with minimal weight updates, leading to lazy learning. Conversely, low-rank initializations force the network to actively construct new features and undergo significant representational shifts.
This perspective is extended by Chi-Ning Chou et al. in Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometry, who demonstrate that low-rank initializations start with poor task separability but actively untangle task-relevant manifolds during training, whereas high-rank initializations follow a conservative trajectory that relies heavily on the initial random state.
The Theoretical vs. Practical Gap
While the mathematical utility of infinitesimal initialization is clear, implementing it in real-world deep learning presents severe engineering challenges.
The authors of Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion, Zhiwei Bai et al., highlight that triggering these non-linear, low-rank dynamics often requires initialization scales so small that they exceed standard floating-point precision. When weights are set too close to zero, models suffer from vanishing gradients, causing the optimization process to stall or fail entirely. This creates a fundamental trade-off between achieving the desired algorithmic bias and maintaining numerical stability during training.
Key Foundational Work
Implicit Regularization in Matrix Factorization (2017) — contributed 2 concept elements to this direction.
Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations (2018) — contributed 2 concept elements to this direction.