Implicit Bias in Non-Homogeneous Neural Networks

This direction is merged into another direction: architectural homogeneity constraint (model homogeneity assumption). Its last version stays readable and citable here.

Version 1 of 146 papers54 conceptsfrozen 2026-10-04

Deep learning works because of a "secret sauce" known as implicit bias. Even when a model has far more parameters than data points—allowing it to fit the training data in an infinite number of ways—gradient descent consistently picks solutions that generalize well.

For years, theorists explained this by treating neural networks as homogeneous functions. In a homogeneous model (like a ReLU network without biases), scaling the weights by a factor $\alpha$ scales the output by $\alpha^k$. This mathematical symmetry allows researchers to prove that gradient descent acts as an implicit regularizer, maximizing the "margin" between classes and balancing the weights across layers.

However, modern architectures like ResNets and Transformers are non-homogeneous. They use skip connections, bias terms, and normalization layers that break these simple scaling rules. Recent research is finally bridging this gap, extending the theory of implicit bias to the complex, "messy" architectures used in the real world.

The Homogeneity Assumption

To understand why non-homogeneity is a challenge, we first have to look at why homogeneity was so useful. In a perfectly homogeneous network, Simon S. Du and Jason D. Lee showed that gradient flow automatically keeps layers "balanced." If you start with small weights, the squared norms of adjacent layers stay synchronized throughout training Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced.

This symmetry makes it easy to prove that the model will converge to a maximum margin solution. Just as a Support Vector Machine (SVM) tries to find the widest possible road between two classes of data, gradient descent in a homogeneous network implicitly pushes the weights toward a state that maximizes this geometric gap.

Why Real Networks Break the Theory

The problem is that almost every component of a state-of-the-art model violates homogeneity:

  • Bias Terms: Adding a simple $+b$ to a layer means that scaling the weights no longer scales the output proportionally.
  • Skip Connections: In a ResNet, the output is a mix of a transformed signal and the original input ($f(x) + x$), which destroys the scaling symmetry.
  • Normalization: LayerNorm and BatchNorm introduce scale-invariant components that behave very differently from standard weights.

As Gal Vardi notes, these non-homogeneous features create a significant gap between theoretical proofs and practical behavior On the Implicit Bias in Deep-Learning Algorithms. When homogeneity is lost, the "auto-balancing" of layers can be disrupted, and the standard proofs for margin maximization no longer strictly apply.

Bridging the Gap with Near-Homogeneity

To fix this, researchers have introduced new mathematical frameworks that relax the strict requirements of homogeneity.

Daniel Kunin and Surya Ganguli proposed the concept of quasi-homogeneous neural networks. They proved that even in models with biases and normalization, gradient flow still exhibits an "asymmetric favoritism" toward specific parameters, which explains phenomena like Neural Collapse—where the final layer's features collapse into a simple geometric structure The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks.

More recently, Yuhang Cai and Jingfeng Wu introduced a near-homogeneity condition. They demonstrated that for a broad class of non-homogeneous networks, gradient descent still maximizes a "normalized margin" Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. This suggests that the "margin-seeking" behavior of neural networks is more robust than previously thought—it doesn't require perfect symmetry to function.

Directional Convergence

One of the most counter-intuitive aspects of training with exponential-tailed losses (like cross-entropy) is that the weights often grow toward infinity. You might expect the model's behavior to become unstable as the weights explode.

However, Mor Shpigel Nacson and Daniel Soudry showed that in both homogeneous and non-homogeneous models, the weights eventually converge in direction Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models. Even if the magnitude of the weights is increasing, the "shape" of the function they represent settles into a stable state. This directional stability is what allows the model to maintain a consistent decision boundary and generalize to new data.

The Path to Grokking

Understanding these biases also helps explain mysterious training behaviors like grokking—where a model suddenly transitions from memorization to perfect generalization long after the training loss has hit zero.

Kaifeng Lyu and Zhiyuan Li argue that grokking is driven by a "dichotomy of implicit biases." In the early phase of training, the model is biased toward complex, overfitting solutions. In the late phase, the margin-maximization bias takes over, slowly pushing the model toward a simpler, more generalizable solution Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking.

By extending these theories to non-homogeneous architectures, researchers are finally uncovering why the complex models we use in practice—from ResNets to Transformers—possess an inherent drive toward simplicity and generalization.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_bias_in_non_homogeneous_neural__387d3bfa,
  title = {Implicit Bias in Non-Homogeneous Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-bias-in-non-homogeneous-neural-networks/rdr_387d3bfa17144a42807a2a8e4962c47a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_implicit_bias_in_non_homogeneous_neural__387d3bfa_v1,
  title = {Implicit Bias in Non-Homogeneous Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-bias-in-non-homogeneous-neural-networks/rdr_387d3bfa17144a42807a2a8e4962c47a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}