The Geometry of Implicit Bias in Optimization

This direction is merged into another direction: implicit max-margin bias. Its last version stays readable and citable here.

Version 1 of 134 papers64 conceptsfrozen 2026-10-04

In modern deep learning, we often train models with far more parameters than data points. In this "overparameterized" regime, there isn't just one set of weights that can perfectly fit the training data—there are infinitely many. Yet, models trained with simple algorithms like Gradient Descent (GD) or Adam generalize remarkably well to unseen data.

This phenomenon is driven by implicit bias: the tendency of an optimization algorithm to favor specific types of solutions over others, even when the loss function doesn't explicitly ask for them. Recent research has revealed that this bias is not accidental; it is a direct consequence of the "geometry" of the optimizer's update rule.

The Geometry of the Step

To understand implicit bias, we must first look at how an optimizer chooses its next move. Every optimization step is a trade-off: we want to decrease the loss as much as possible without moving too far from our current position.

Jason D. Lee, Suriya Gunasekar, and colleagues established in Characterizing Implicit Bias in Terms of Optimization Geometry that the definition of "far" determines the final solution. If we measure distance using the standard Euclidean ($\ell_2$) norm, we get Gradient Descent. If we use a different norm or a non-Euclidean "potential function," we get a different algorithm entirely, such as Mirror Descent.

Nikolaos Tsilivis, Gal Vardi, and Julia Kempe expanded this into a unifying theory for homogeneous neural networks in Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks. They proved that an optimizer using a specific norm $\|\cdot\|$ to define its "steepest" direction will implicitly maximize a geometric margin corresponding to that same norm.

Why Adam and SignGD Behave Differently

Practitioners have long noted that adaptive optimizers like Adam and Sign-based Gradient Descent (SignGD) behave differently than standard GD. Theoretical work by Chenyang Zhang, Difan Zou, and Yuan Cao in The Implicit Bias of Adam on Separable Data shows that while GD is biased toward the $\ell_2$ maximum-margin solution (like a Support Vector Machine), Adam and SignGD are biased toward the $\ell_\infty$ maximum-margin solution.

This shift from a spherical ($\ell_2$) to a box-shaped ($\ell_\infty$) geometry has profound implications:

The Horizon of Potential Functions

For more complex optimizers like Mirror Descent, the bias is determined by a "potential function." Scott Pesme and Radu-Alexandru Dragomir introduced the concept of the horizon function in Implicit Bias of Mirror Flow on Separable Data.

As training progresses and weights grow toward infinity, the local curvature of the potential function matters less than its asymptotic shape—the "horizon." This horizon function defines the specific norm that the model eventually maximizes. This framework allows researchers to "engineer" implicit biases by choosing potentials whose horizons correspond to desired properties, such as sparsity or low-rank structure. For example, Chen Fan and Christos Thrampoulidis showed that the Muon optimizer is biased toward maximizing the margin relative to the spectral norm, explaining its success in large-scale matrix-heavy training (Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data).

The Chameleon Effect of Batch Size

While the optimizer's rule sets the stage, the training regime can shift the final outcome. Beomhan Baek and Chulhee Yun discovered that Adam's implicit bias is a "chameleon."

In Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime, they show that while full-batch Adam favors $\ell_\infty$ solutions, using small mini-batches can shift the bias toward a data-dependent Mahalanobis norm or even revert it to the standard $\ell_2$ bias of GD. This suggests that the "Adam advantage" in generalization might be sensitive to the batch size used during training.

Engineering Inductive Bias

By viewing optimization as a geometric choice, researchers are moving from observing implicit bias to designing it.

  • Path-SGD: Behnam Neyshabur and Nathan Srebro proposed Path-SGD, which uses a path-normalized geometry to make optimization invariant to weight rescaling, leading to better generalization in ReLU networks.
  • Reflectron: Nicholas M. Boffi and Jean-Jacques E. Slotine showed that matching the optimization geometry to the data's structure (e.g., using sparse potentials for sparse data) can reduce generalization error from polynomial to logarithmic scales (The role of optimization geometry in single neuron learning).

Ultimately, this research direction suggests that the optimizer is not just a tool to find a minimum; it is a fundamental part of the model's architecture that dictates how it perceives and represents the world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_geometry_of_implicit_bias_in_optimiz_acafef8a,
  title = {The Geometry of Implicit Bias in Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-geometry-of-implicit-bias-in-optimization/rdr_acafef8ac466443fae7a0421941a0c3d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_geometry_of_implicit_bias_in_optimiz_acafef8a_v1,
  title = {The Geometry of Implicit Bias in Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-geometry-of-implicit-bias-in-optimization/rdr_acafef8ac466443fae7a0421941a0c3d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}