The Implicit Bias of Gradient Based Optimization

This direction is merged into another direction: overparameterization generalization paradox. Its last version stays readable and citable here.

Version 1 of 177 papers86 conceptsfrozen 2026-10-04

In classical statistics, the recipe for a well-behaved model is simple: do not use more parameters than you have data points. If you violate this rule, your model will overfit, memorizing the noise in your training set and failing to generalize to new examples.

Deep learning violates this rule spectacularly. Modern neural networks often have millions or billions more parameters than training samples. Yet, as Chiyuan Zhang, Benjamin Recht, and colleagues famously demonstrated in Understanding deep learning requires rethinking generalization, these models generalize excellently on real data even though they are perfectly capable of memorizing random noise.

This suggests that the "real" inductive bias—the reason these models work—doesn't come from limiting the number of parameters. Instead, it comes from implicit regularization: a hidden preference for simple solutions that is baked into the optimization process itself.

Optimization is Not a Neutral Tool

When we train a model, we usually think of the optimizer (like Stochastic Gradient Descent) as a neutral tool that simply finds the lowest point on a loss landscape. However, in an overparameterized regime, there isn't just one "lowest point." There is an entire manifold of solutions that achieve zero training error.

The breakthrough insight from researchers like Behnam Neyshabur and Nathan Srebro is that the path the optimizer takes matters as much as the destination. In In Search of the Real Inductive Bias, they argue that gradient descent acts as an implicit regularizer, steering the model toward solutions with small "norms" (a measure of complexity) even when no explicit penalty like weight decay is used.

The Geometry of Simplicity

Why does the optimizer prefer simple solutions? One reason is the geometry of the loss landscape.

Imagine a landscape with two types of valleys: one is a narrow, sharp "crevice," and the other is a wide, "flat" basin. Both might have the same depth (zero training loss), but the flat basin is more robust. If you move a tiny bit away from the solution (as you might when switching from training data to test data), the loss in the flat basin stays low, while the loss in the sharp crevice spikes.

David G. T. Barrett and Benoit Dherin showed in Implicit Gradient Regularization that the discrete nature of gradient descent—taking steps of a certain size rather than moving continuously—naturally penalizes steep gradients. This "hidden" penalty pushes the model away from sharp, complex minima and toward the flat, smooth regions that generalize better.

Simplicity is More Likely

There is also a structural reason for this bias. In Deep learning generalizes because the parameter-function map is biased towards simple functions, Guillermo Valle-Pérez and Ard A. Louis apply Algorithmic Information Theory to show that "simple" functions occupy a much larger volume of the parameter space than "complex" ones.

Think of it like throwing a dart at the space of all possible neural network weights. You are exponentially more likely to hit a set of weights that represents a simple, smooth function than one that represents a chaotic, noisy one. Because the optimizer starts with small weights and moves toward the nearest solution, it is almost guaranteed to find one of these "simple" functions first.

Benign Overfitting and Grokking

This implicit bias explains two of the most counter-intuitive phenomena in modern AI: benign overfitting and grokking.

  1. Benign Overfitting: Peter L. Bartlett and Alexander Rakhlin have explored how models can perfectly fit noisy data without ruining their performance on clean data. The optimizer fits the "signal" using the most efficient, low-norm parts of the model, while the "noise" is relegated to high-frequency components that don't interfere with the overall prediction.
  2. Grokking: As Wei Hu and others have noted, models sometimes "overfit" a task for a long time before suddenly "grokking" the underlying rule. This happens because the implicit regularization of gradient descent slowly but surely whittles away the complex, memorized solution in favor of a simpler, more generalizable one that eventually "wins" the competition for the model's weights.

The Collapse to Structure

In the final stages of training, this bias toward simplicity manifests as Neural Collapse. Ebrahim Rasromani describes this as a state where the network's internal representations simplify into highly symmetric geometric structures.

Similarly, Spencer Frei and Gal Vardi found that leaky ReLU networks trained on high-dimensional data are biased toward extremely low-rank solutions. Even though the network could use its millions of parameters to create a jagged, complex decision boundary, the optimization process forces it to act like a much simpler, linear model.

By understanding that the optimizer is not just a solver but a filter, we can see why deep learning works: it is a system that naturally prefers the simplest explanation that fits the data, effectively rediscovering Ockham's Razor through the mathematics of gradient flow.

Papers in this version

Showing 60 of 77 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_implicit_bias_of_gradient_based_opti_4cb42696,
  title = {The Implicit Bias of Gradient Based Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-implicit-bias-of-gradient-based-optimization/rdr_4cb4269613ab4d4d86ae74481a52801f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_implicit_bias_of_gradient_based_opti_4cb42696_v1,
  title = {The Implicit Bias of Gradient Based Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-implicit-bias-of-gradient-based-optimization/rdr_4cb4269613ab4d4d86ae74481a52801f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}