The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks

Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli

This paper provides a significant theoretical leap in understanding why modern neural networks generalize. By moving beyond idealized "homogeneous" models, the authors explain the implicit bias of the complex, heterogeneous architectures actually used in production today.

The Generalization Paradox

Deep learning presents a well-known paradox: networks possess enough capacity to memorize random noise, yet when trained on real data, they consistently find solutions that generalize remarkably well. Researchers have long theorized that an implicit bias in gradient descent acts as a hidden regularizer, pushing the model toward "simple" or "max-margin" solutions.

However, most existing theoretical frameworks only apply to homogeneous networks—such as simple ReLU networks without biases or skip connections. In these idealized models, scaling all parameters by a factor $\alpha$ scales the output uniformly by $\alpha^L$. Real-world architectures like ResNets or Transformers are not homogeneous. They contain biases, residual branches, and normalization layers that scale at entirely different rates. This paper bridges the gap between theory and practice by defining a broader class of models and identifying their specific implicit bias.

Understanding Quasi-Homogeneous Models

To address this gap, the authors introduce the class of $\Lambda$-Quasi-Homogeneous models. In these models, different parameters can have different "scaling rates" (denoted as $\lambda$). If you scale a parameter $\theta_i$ by $e^{\alpha \lambda_i}$, the entire model output scales by $e^\alpha$.

Contour plots illustrating symmetric (homogeneous) vs. asymmetric (quasi-homogeneous) scaling rates.

This framework is highly expressive and can accurately describe:

  • Fully connected networks with biases.
  • ResNets, where skip connections create computational paths of varying lengths.
  • Networks with Batch or Layer Normalization.

The researchers analyzed the gradient flow (the continuous-time version of gradient descent) of these models using an exponential loss function, such as cross-entropy. Using geometric analysis, they tracked how the "angle" of the parameters evolves relative to the "characteristic curves" of the model's scaling properties.

Visualizing the Bias in Action

To validate their proofs, the authors utilized synthetic datasets, including linearly separable Gaussians and a 3D "Ball Classification" task. These experiments highlight how quasi-homogeneous scaling alters the decision boundary.

Different parameterizations of logistic regression lead to distinct decision boundaries.

Key Insights: The "Highest-Rate" Favoritism

The core discovery of the paper is that gradient flow is effectively "lazy" regarding parameters that scale slowly. It focuses almost entirely on minimizing the norm of the highest-rate parameters—those with the largest $\lambda$.

  • Asymmetric Norms: Unlike homogeneous networks where the "total" norm is minimized equally, quasi-homogeneous networks minimize an asymmetric norm. For a ResNet, this implies the implicit bias primarily "cares" about the weights in the very last layer, effectively ignoring the complexity of earlier layers during the final stages of training.
  • Robustness vs. Sparsity: This favoritism is a double-edged sword. It can help the model discard unnecessary parameters, leading to a sparser representation. However, it can also lead to solutions with a very small margin in the input space, making the model fragile to small perturbations (such as adversarial attacks). In their Ball Classification experiment, the authors demonstrated that a quasi-homogeneous model's robustness drops sharply compared to a homogeneous model because the implicit bias ignores the parameters needed to maintain a "thick" margin.
  • Explaining Neural Collapse: "Neural Collapse" is an empirical phenomenon where, at the end of training, last-layer features of the same class collapse to a single point, forming a symmetric structure (a simplex ETF). The authors prove that their asymmetric max-margin theorem perfectly predicts this behavior. For any sufficiently expressive network with a normalization layer, the last-layer weights and biases are the "highest-rate" parameters. Their theorem dictates that these parameters must reach the Neural Collapse state to satisfy the max-margin condition.

Main Theoretical Result and Limitations

The paper's central theorem formally proves that for quasi-homogeneous models, the parameters converge in direction to a first-order KKT point of a maximum-margin problem. Crucially, the objective of this optimization is only the norm of the parameters with the maximum scaling rate ($\lambda_{max}$).

While groundbreaking, the authors note a few limitations:

  1. Gradient Flow vs. SGD: The proof assumes continuous gradient flow. Real-world training uses discrete steps (SGD), and finite learning rates can introduce "discretization effects" that might alter the bias.
  2. Initialization Dependency: The analysis focuses on the "terminal phase" of training (after the data is separated). The model's initialization determines whether it ever reaches this regime.
  3. Local vs. Global Optimality: The theorem guarantees convergence to a KKT point (a local optimum), but does not guarantee that gradient descent will find the global maximum-margin solution.

Ultimately, this work provides a vital mathematical foundation for understanding why modern, complex neural networks behave the way they do, offering a unified explanation for phenomena ranging from robustness decay to Neural Collapse.

Summary updated

Abstract

In this work, we explore the maximum-margin bias of quasi-homogeneous neural networks trained with gradient flow on an exponential loss and past a point of separability. We introduce the class of quasi-homogeneous models, which is expressive enough to describe nearly all neural networks with homogeneous activations, even those with biases, residual connections, and normalization layers, while structured enough to enable geometric analysis of its gradient dynamics. Using this analysis, we generalize the existing results of maximum-margin bias for homogeneous networks to this richer class of models. We find that gradient flow implicitly favors a subset of the parameters, unlike in the case of a homogeneous model where all parameters are treated equally. We demonstrate through simple examples how this strong favoritism toward minimizing an asymmetric norm can degrade the robustness of quasi-homogeneous models. On the other hand, we conjecture that this norm-minimization discards, when possible, unnecessary higher-order parameters, reducing the model to a sparser parameterization. Lastly, by applying our theorem to sufficiently expressive neural networks with normalization layers, we reveal a universal mechanism behind the empirical phenomenon of Neural Collapse.
BibTeX
@inproceedings{kunin2023asymmetric,
  author = {Daniel Kunin and Atsushi Yamamura and Chao Ma and Surya Ganguli},
  title = {{The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=IM4xp7kGI5V},
}
Figure