Simplicity Bias via Global Convergence of Sharpness Minimization

Khashayar Gatmiry, Zhiyuan Li, Sashank J. Reddi, Stefanie Jegelka

Introduction

A central paradox in deep learning is why models with far more parameters than training examples consistently generalize to unseen data instead of simply memorizing noise. This phenomenon is widely attributed to the implicit bias of Stochastic Gradient Descent (SGD). Historically, researchers have approached this bias through two distinct lenses: simplicity bias, where SGD favors functions with low-rank feature representations, and sharpness minimization, where the optimizer gravitates toward "flat" regions of the loss landscape.

While an intuitive link between flatness and simplicity has long been suspected, a formal mathematical bridge has remained elusive for non-linear networks. Furthermore, most existing theoretical results are limited to local convergence, describing model behavior only when it is already near a solution. This paper provides a rigorous theoretical framework proving that for two-layer networks, sharpness minimization directly implies simplicity bias and that this convergence occurs globally from any initialization.

The Mechanism: Label Noise SGD

The authors analyze the dynamics of label noise SGD, a training variant that adds Gaussian noise to the targets. This noise acts as a regularizer that pushes the model away from "sharp" minima toward flatter regions. The study focuses on two-layer neural networks utilizing specific activation functions, such as the cubic activation $x^3$, which possess properties like positivity and convexity in their derivatives.

Mathematical Framework

To track the model's trajectory, the researchers employ three key technical approaches:

  1. Manifold Analysis: They study the "manifold of zero loss," representing all parameter configurations that perfectly fit the training data.
  2. Riemannian Gradient Flow: The trajectory of label noise SGD is modeled as a flow moving along this manifold, driven by the gradient of the loss Hessian's trace (a proxy for sharpness).
  3. Local Geodesic Convexity (g-convexity): The authors identify that while the sharpness objective is non-convex in standard Euclidean space, it becomes convex when analyzed through the lens of the manifold's geometry near stationary points. This "secret sauce" allows them to prove global convergence.

Key Insights: The Rank-One Collapse

The most significant finding of the paper is the rank-one collapse. The authors demonstrate that at the global minimum of sharpness, all neurons in a two-layer network collapse to replicate a single linear feature. This results in a feature matrix with a rank of exactly one—the simplest possible configuration that can satisfy the training constraints.

This finding mathematically formalizes the idea that minimizing the trace of the Hessian (sharpness) is equivalent to forcing the model to find the lowest-rank representation possible. In the high-dimensional regime (where the input dimension $d$ is greater than or equal to the number of data points $n$), this collapse is total. In lower-dimensional regimes ($d < n$), the authors observe neuron clustering, where neurons group into a small number of clusters to minimize complexity even when a rank-one solution is insufficient to fit the data.

Theoretical and Empirical Results

The paper provides the first proof of global linear convergence for this process. Specifically, Theorems 1.1 and 3.5 formalize that label noise SGD, given a sufficiently small step size, will reach the manifold of zero loss and then move toward the global minimizer of sharpness at an exponential rate. This is a significant advancement over previous "local" results that only characterized behavior in the final stages of training.

Empirical validations using synthetic datasets support these theorems. In tests with high-dimensional inputs, the second and third singular values of the feature matrix were shown to crash to zero during training, leaving a single dominant singular value. This confirms that the resulting model's functional form becomes remarkably simple: a single-variable function applied to a linear projection of the input.

Limitations and Future Directions

While the paper establishes a robust link between sharpness and simplicity, it identifies several avenues for future research:

  • Architectural Depth: The rigorous proofs are currently confined to two-layer networks. While the authors hypothesize that these principles extend to deeper architectures, the mathematical complexity increases significantly.
  • Activation Functions: The results rely on specific derivative properties. While many common activations are covered, non-smooth functions like ReLU require further approximation analysis.
  • Dimensionality: The cleanest "rank-one" results assume $d \ge n$. Extending the theory to more thoroughly explain the clustering behavior observed in the $d < n$ regime remains a task for future work.

In summary, this work provides a foundational theoretical bridge, showing that the search for flat minima is fundamentally a search for simplicity, and that this search is globally effective in the optimization landscape of two-layer networks.

Summary updated

BibTeX
@inproceedings{gatmiry2024simplicity,
  author = {Khashayar Gatmiry and Zhiyuan Li and Sashank J. Reddi and Stefanie Jegelka},
  title = {{Simplicity Bias via Global Convergence of Sharpness Minimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=VUTyzH63Xa},
}
Figure