In classical statistics, a model with more parameters than data points is a recipe for disaster. Such "overparameterized" models should, in theory, perfectly memorize the training noise and fail to generalize to anything new. Yet, modern deep learning thrives in this regime.
The mystery is not that these models can fit the data—it is that among the infinite number of ways to fit the data perfectly, the optimization process consistently picks the ones that generalize. This phenomenon is known as implicit bias: the hidden preference of training algorithms like Gradient Descent (GD) for "simple" solutions, even when no explicit regularization (like weight decay) is used.
The Geometry of Simplicity
When a network is trained on separable data using a classification loss (like cross-entropy), it doesn't just stop once it reaches zero training error. Instead, the weights continue to grow in magnitude to push the training loss even closer to zero.
Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro have shown in Lexicographic and Depth-Sensitive Margins and Convergence of Gradient Descent on Separable Data that this late-stage training has a specific geometric effect. Gradient descent acts as an implicit margin maximizer. Much like a Support Vector Machine (SVM), it biases the network toward the solution that maximizes the distance between the decision boundary and the nearest data points.
This bias isn't limited to simple models. In deep linear networks, Matus Telgarsky proved in Gradient descent aligns the layers of deep linear networks that GD forces a unique internal coordination. Every layer's weight matrix eventually simplifies into a rank-1 matrix, and these matrices "align" their directions with one another. This internal simplification is what ultimately drives the entire network toward a maximum-margin solution.
Architecture as a Filter for Bias
While the optimizer provides the "push" toward simplicity, the architecture of the network determines what "simple" actually looks like.
In Implicit Bias of Gradient Descent on Linear Convolutional Networks, Jason D. Lee and colleagues demonstrated that the structural constraints of a network—like weight sharing in convolutions—fundamentally change the bias. While a fully connected network might prefer a standard $\ell_2$ margin, a convolutional network is biased toward sparsity in the frequency domain.
This suggests a "disentanglement" of effects. Amit Peleg and Matthias Hein argue in Bias of Stochastic Gradient Descent or the Architecture that increasing network width primarily helps generalization by giving the optimizer more room to express its implicit bias. In contrast, increasing depth can sometimes be detrimental if the architecture's inherent properties conflict with the optimizer's path.
The Dynamics of Grokking
One of the most striking examples of implicit bias in action is grokking—the phenomenon where a model suddenly transitions from 0% to 100% test accuracy long after it has already achieved 100% training accuracy.
Kaifeng Lyu, Zhiyuan Li, and Jason D. Lee explain this in Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking as a competition between two different biases:
- Early Phase: The model is biased toward "kernel" solutions, which effectively memorize the training data using the easiest available features.
- Late Phase: A slower, "max-margin" bias takes over, gradually reshaping the weights into a more generalizable form.
The "aha!" moment of grokking is simply the point where the late-phase, generalizable bias finally overcomes the early-phase memorization.
Beyond the Optimizer
Is generalization only about the optimizer? Not necessarily. Some researchers suggest that the map from parameters to functions is itself biased. Guillermo Valle-Pérez and Ard A. Louis argue in Deep learning generalizes because the parameter-function map is biased towards simple functions that simple functions occupy an exponentially larger volume of the parameter space. In this view, GD finds simple solutions partly because they are "everywhere" in the landscape.
Furthermore, David Gamarnik and Ilias Zadik have shown that for certain architectures, like two-layer networks with non-negative weights, low training error automatically implies a small output norm. This "self-regularity" (Self-Regularity of Non-Negative Output Weights) suggests that for some models, generalization is baked into the architecture and data distribution themselves, regardless of the specific algorithm used.
Why It Works
The reason overparameterized networks work is that they are never truly "free" to explore their entire capacity. From the moment of initialization—where the "shape" of the starting weights sets the initial trajectory (Moroshko et al.)—to the final steps of convergence, the optimization process is constrained.
Whether it is the alignment of layers in deep networks, the maximization of geometric margins, or the inherent simplicity of the parameter-to-function map, the "implicit" part of deep learning is what prevents the "overparameterized" part from falling into chaos. Understanding these biases allows us to move from treating neural networks as black boxes to seeing them as precision instruments that naturally gravitate toward the most useful solutions.