In modern deep learning, we often train models with far more parameters than data points. In this "overparameterized" regime, there isn't just one set of weights that can perfectly fit the training data—there are infinitely many. Yet, models trained with simple algorithms like Gradient Descent (GD) or Adam generalize remarkably well to unseen data.
This phenomenon is driven by implicit bias: the tendency of an optimization algorithm to favor specific types of solutions over others, even when the loss function doesn't explicitly ask for them. Recent research has revealed that this bias is not accidental; it is a direct consequence of the "geometry" of the optimizer's update rule.
The Geometry of the Step
To understand implicit bias, we must first look at how an optimizer chooses its next move. Every optimization step is a trade-off: we want to decrease the loss as much as possible without moving too far from our current position.
Jason D. Lee, Suriya Gunasekar, and colleagues established in Characterizing Implicit Bias in Terms of Optimization Geometry that the definition of "far" determines the final solution. If we measure distance using the standard Euclidean ($\ell_2$) norm, we get Gradient Descent. If we use a different norm or a non-Euclidean "potential function," we get a different algorithm entirely, such as Mirror Descent.
Nikolaos Tsilivis, Gal Vardi, and Julia Kempe expanded this into a unifying theory for homogeneous neural networks in Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks. They proved that an optimizer using a specific norm $\|\cdot\|$ to define its "steepest" direction will implicitly maximize a geometric margin corresponding to that same norm.
Why Adam and SignGD Behave Differently
Practitioners have long noted that adaptive optimizers like Adam and Sign-based Gradient Descent (SignGD) behave differently than standard GD. Theoretical work by Chenyang Zhang, Difan Zou, and Yuan Cao in The Implicit Bias of Adam on Separable Data shows that while GD is biased toward the $\ell_2$ maximum-margin solution (like a Support Vector Machine), Adam and SignGD are biased toward the $\ell_\infty$ maximum-margin solution.
This shift from a spherical ($\ell_2$) to a box-shaped ($\ell_\infty$) geometry has profound implications:
- Feature Diversity: Bhavya Vasudeva and Mahdi Soltanolkotabi argue in The Rich and the Simple: On the Implicit Bias of Adam and SGD that Adam is more resistant to "simplicity bias." While SGD might latch onto a few easy-to-learn features, Adam's $\ell_\infty$ bias encourages the model to utilize a "richer," more distributed set of features.
- Heavy-Tailed Data: In language modeling, where data follows a heavy-tailed distribution (Zipf’s Law), Robin Yadav and Zhiyuan Li demonstrate that Sign-based descent converges exponentially faster than GD because its coordinate-wise updates better match the geometry of the softmax loss landscape (Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance).
The Horizon of Potential Functions
For more complex optimizers like Mirror Descent, the bias is determined by a "potential function." Scott Pesme and Radu-Alexandru Dragomir introduced the concept of the horizon function in Implicit Bias of Mirror Flow on Separable Data.
As training progresses and weights grow toward infinity, the local curvature of the potential function matters less than its asymptotic shape—the "horizon." This horizon function defines the specific norm that the model eventually maximizes. This framework allows researchers to "engineer" implicit biases by choosing potentials whose horizons correspond to desired properties, such as sparsity or low-rank structure. For example, Chen Fan and Christos Thrampoulidis showed that the Muon optimizer is biased toward maximizing the margin relative to the spectral norm, explaining its success in large-scale matrix-heavy training (Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data).
The Chameleon Effect of Batch Size
While the optimizer's rule sets the stage, the training regime can shift the final outcome. Beomhan Baek and Chulhee Yun discovered that Adam's implicit bias is a "chameleon."
In Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime, they show that while full-batch Adam favors $\ell_\infty$ solutions, using small mini-batches can shift the bias toward a data-dependent Mahalanobis norm or even revert it to the standard $\ell_2$ bias of GD. This suggests that the "Adam advantage" in generalization might be sensitive to the batch size used during training.
Engineering Inductive Bias
By viewing optimization as a geometric choice, researchers are moving from observing implicit bias to designing it.
- Path-SGD: Behnam Neyshabur and Nathan Srebro proposed Path-SGD, which uses a path-normalized geometry to make optimization invariant to weight rescaling, leading to better generalization in ReLU networks.
- Reflectron: Nicholas M. Boffi and Jean-Jacques E. Slotine showed that matching the optimization geometry to the data's structure (e.g., using sparse potentials for sparse data) can reduce generalization error from polynomial to logarithmic scales (The role of optimization geometry in single neuron learning).
Ultimately, this research direction suggests that the optimizer is not just a tool to find a minimum; it is a fundamental part of the model's architecture that dictates how it perceives and represents the world.