The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli

Introduction

A central mystery in deep learning is why over-parameterized neural networks generalize so effectively despite having the capacity to simply memorize training data. The prevailing explanation is implicit bias: the optimization algorithm (typically Gradient Descent) does not merely find any solution that fits the data, but is naturally "pushed" toward solutions with specific geometric properties, such as those that maximize the classification margin.

However, traditional mathematical proofs for implicit bias often rely on the assumption of "homogeneity"—where scaling all weights by a factor $\alpha$ scales the output by $\alpha^L$. Real-world architectures like ResNets, or those employing bias terms and Batch Normalization, break this simple symmetry. This paper introduces a new framework, $\Lambda$-Quasi-Homogeneous models, to bridge the gap between clean theory and the complex architectures used in practice, revealing a phenomenon termed "asymmetric favoritism."

The $\Lambda$-Quasi-Homogeneous Framework

The authors define $\Lambda$-Quasi-Homogeneous models as those where different parameters can scale at different rates. Specifically, if parameter $i$ is scaled by $e^{\alpha \lambda_i}$, the total output scales by $e^\alpha$. This definition encompasses a broad range of modern architectures, including networks with skip connections and normalization layers.

Visualization of the asymmetric norm geometry in parameter space $\theta_1, \theta_2$.

By analyzing gradient flow on exponential losses (like cross-entropy) within this framework, the authors track how parameters evolve. They prove that as training progresses past the point of separability, the network is driven toward a maximum-margin solution. Crucially, this solution minimizes an asymmetric norm that prioritizes parameters with higher scaling rates $\lambda_i$.

The Mechanism of Asymmetric Favoritism

Unlike homogeneous networks, where every weight is penalized equally by the implicit bias, quasi-homogeneous networks exhibit "favorites." Gradient flow focuses on minimizing the norm of the "highest-rate" parameters—those that contribute most significantly to the scaling of the output—while effectively ignoring others.

Comparison of implicit bias in standard (a) vs. quasi-homogeneous (b) parameterizations. The solid line represents the learned boundary, while the dashed line is the standard max-margin solution.

As shown in the figure above, different parameterizations of the same underlying model can lead to vastly different decision boundaries. In a standard linear model (a), the learned boundary aligns with the expected maximum margin. However, in a quasi-homogeneous parameterization (b), the "asymmetric favoritism" causes the boundary to shift, prioritizing the margin relative to a specific subset of parameters. This can be a double-edged sword: while it may act as a form of implicit pruning, it can also lead to a robustness collapse, where the model chooses a boundary dangerously close to the data in "lower-rate" directions.

Implications: Neural Collapse and Robustness

The paper provides a theoretical explanation for Neural Collapse, the empirical observation that last-hidden-layer features of the same class collapse to a single point (the class mean) during the terminal phase of training. The authors prove that Neural Collapse is the mathematically inevitable result of this asymmetric maximum-margin bias in networks with normalization layers.

Furthermore, the theory predicts that robustness is not guaranteed. In synthetic "Ball Classification" experiments, the authors demonstrated that while a standard homogeneous model maintains a steady margin, a quasi-homogeneous model's robustness can degrade as the data geometry changes, exactly as predicted by the asymmetric bias toward high-rate parameters.

Conclusion and Future Work

This work generalizes the theory of implicit bias to a much wider and more realistic class of neural networks. By proving that gradient flow converges to a KKT point of an asymmetric max-margin problem (Theorem 4.1), the authors provide a rigorous foundation for understanding why modern networks behave the way they do.

Future research is needed to extend these findings from continuous gradient flow to discrete gradient descent, where the learning rate may introduce additional biases. Additionally, exploring how the noise of Stochastic Gradient Descent (SGD) and different initialization regimes (kernel vs. rich) interact with this asymmetric favoritism remains an open and vital area of study.

Summary updated

BibTeX
@inproceedings{kunin2023asymmetric,
  author = {Daniel Kunin and Atsushi Yamamura and Chao Ma and Surya Ganguli},
  title = {{The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=IM4xp7kGI5V},
}
Figure