In classical statistics, a model with more parameters than data points is expected to overfit—to "memorize" the noise rather than learn the underlying pattern. Yet, modern deep learning thrives in this overparameterized regime. The mystery is not that these models can generalize, but why they choose to.
Recent research suggests the answer lies in implicit bias: the tendency of optimization algorithms like Gradient Descent (GD) to favor specific, "simple" solutions among the infinite number of parameter configurations that perfectly fit the training data.
The Geometry of the Max-Margin Solution
The foundational intuition for implicit bias comes from classification on separable data. When classes can be perfectly split by a hyperplane, there are infinitely many such planes. However, Daniel Soudry, Nathan Srebro, and colleagues demonstrated in The Implicit Bias of Gradient Descent on Separable Data that GD on logistic or cross-entropy loss doesn't just find any separator; it asymptotically converges to the maximum-margin solution.
Why does this happen? As the training loss approaches zero, the weights must grow toward infinity to satisfy the exponential tail of the loss function. In this "late phase" of training, the direction of the weight vector becomes increasingly aligned with the direction that maximizes the distance to the nearest data points—the same solution sought by a hard-margin Support Vector Machine (SVM). This explains why continuing to train after reaching zero training error, as noted by Mor Shpigel Nacson in Convergence of Gradient Descent on Separable Data, can still improve generalization: the model is refining its margin.
Depth as a Rank Minimizer
While the max-margin bias explains linear models, deep networks introduce a different kind of structural bias. In multi-layer architectures, the weights of different layers interact multiplicatively. Nadav Cohen and Noam Razin showed in Implicit Regularization in Tensor Factorization that this multiplicative structure acts as a "greedy" rank minimizer.
As the network gets deeper, this effect intensifies. Arthur Jacot proved in Implicit Bias of Large Depth Networks that infinite depth forces the network to represent functions with low nonlinear rank, effectively compressing data through low-dimensional bottlenecks. Similarly, Baekrok Shin found that depth promotes low-rankness in matrix completion tasks, allowing deep models to recover structures from "disconnected" data where shallow models fail.
Optimization Dynamics and the Rich Regime
The specific path taken by the optimizer—governed by hyperparameters like step size and initialization—also dictates the strength of the bias.
- Initialization Scale: Small-scale initialization tends to put the network in a "rich" regime where it learns complex, low-rank features. Large-scale initialization can lead to the "lazy" regime (the Neural Tangent Kernel regime), where the network behaves like a simple linear model. Aditya Vardhan Varre and Nicolas Flammarion characterize this as a spectral bias toward discovering the most important dimensions of the data.
- Step Size: Large learning rates aren't just for speed. Mor Shpigel Nacson and Kavya Ravichandran showed in Implicit Bias of the Step Size in Linear Diagonal Neural Networks that large step sizes near the "edge of stability" actually bias the model toward sparser solutions.
- Stochastic Noise: The inherent noise in Stochastic Gradient Descent (SGD) acts as a regularizer. Maksym Andriushchenko demonstrated that high learning rates before decay allow SGD to "jump" from complex, high-rank solutions to simpler, low-rank ones—a one-way transition that Zihan Wang and Arthur Jacot formalize as a fundamental property of the optimization trajectory.
The Non-linear Frontier
Despite these advances, a significant theoretical gap remains. Most rigorous proofs for implicit bias—whether toward max-margin solutions or low-rank structures—rely on linear or "deep linear" (multi-layer but without activations) approximations.
As noted across several works, including Meena Jagadeesan and Suriya Gunasekar in their study of Linear Convolutional Networks, extending these guarantees to non-linear activations like ReLU or Tanh is the current "grand challenge." While empirical evidence suggests that non-linear networks follow similar patterns of structural simplification, the mathematical bridge between the non-convex landscape of deep learning and the clean geometry of margin maximization is still being built.