Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

Benjamin Bowman, Guido Montúfar

Introduction

In recent years, the Neural Tangent Kernel (NTK) framework has revolutionized our theoretical understanding of deep learning. It demonstrates that in the limit of infinite width, neural networks behave like linear models with a constant kernel. However, this theory predominantly relies on the "heavily overparameterized" regime, where the network possesses far more parameters than data points.

This paper bridges the theoretical gap between the infinite-width world of NTKs and the practical reality of finite, underparameterized neural networks. In finite networks, the NTK is not constant; it evolves as the weights update during training. Previous theories struggled to account for this "noise" in the kernel's trajectory without assuming massive network width. By introducing the concept of "Damped Deviations," the authors prove that even small networks exhibit a "spectral bias"—learning the most important, simple components of a target function first because errors in the network's evolution are mathematically suppressed for those specific directions.

The "Damped Deviations" Framework

To understand how underparameterized networks optimize Mean Squared Error (MSE), the authors analyze the gradient flow of a one-hidden-layer neural network in function space. Rather than tracking individual weights, they observe how the output function $f(x)$ evolves over time.

The core mathematical innovation is the Damped Deviations Lemma. The authors decompose the residual error of the network during training into two distinct parts:

  1. The Ideal Trajectory: The theoretical path an infinite-width network would take.
  2. The Deviation Term: The error introduced by the fact that the network is finite and its kernel is actively changing.

Crucially, they demonstrate that this deviation term is multiplied by a damping factor directly related to the eigenvalues of the NTK. In directions where the eigenvalue is large—which correspond to "simple" or "low-frequency" components of the data—the damping is strongest. Consequently, the network can learn these fundamental components accurately, even if the underlying kernel is fluctuating wildly.

Theoretical Setup

As a rigorous theoretical work, the paper steps away from standard empirical benchmarks like ImageNet or CIFAR-10, opting instead for precise mathematical environments:

  • Synthetic Data on the Sphere: The analysis focuses on data uniformly distributed on the $d$-dimensional sphere ($S^{d-1}$).
  • Spherical Harmonics: For this specific data distribution, the eigenfunctions of the NTK are spherical harmonics (the higher-dimensional analog of the Fourier basis). This property allows the authors to strictly define and separate "low-frequency" from "high-frequency" components.

Key Findings and Insights

Spectral Bias is Universal

The tendency of neural networks to fit low-frequency components before memorizing high-frequency noise is not merely a quirk of massive models. The authors prove (Theorem 3.5) that this spectral bias is a fundamental property of MSE gradient optimization that holds firmly even in underparameterized settings. The network learns the eigenfunctions of the NTK integral operator at rates proportional to their eigenvalues.

The Damping Effect

The most surprising insight is that the "noise" inherent in a finite kernel's evolution does not hinder the learning of primary features. The mathematics of gradient flow essentially "filters out" the kernel's instability for the directions that matter most (those with large eigenvalues).

Redefining Width Requirements

The paper suggests a paradigm shift in how we view network size. The required width ($m$) of a network should not scale solely as a function of the number of samples ($n$), but rather as a function of the complexity of the target function. If the goal is only to learn the "simple" parts of a function (a compact representation composed of top eigenfunctions), a much narrower network is sufficient.

Improved Overparameterization Bounds

Beyond the underparameterized regime, the damped deviations perspective allows the authors to tighten existing bounds for overparameterized networks. They provide stricter requirements for width than previous landmark papers (e.g., Arora et al., 2019), successfully reducing the required width by a factor of $n^2$ in certain settings.

Limitations and Future Directions

While the Damped Deviations framework offers a powerful new lens for understanding network dynamics, the authors note several limitations that pave the way for future research:

  • Activation Functions: The current proofs rely on "smooth" activation functions (such as Sigmoid or Softplus). While approximations for the popular ReLU activation are discussed, a direct proof for non-smooth functions remains an open challenge.
  • Architectural Depth: The analysis is primarily restricted to shallow, one-hidden-layer networks. Extending this framework to deep architectures is a logical and necessary next step.
  • Loss Function Specificity: The theoretical guarantees are specific to Mean Squared Error (MSE). It remains unclear if a similar damping mechanism exists for Cross-Entropy loss, which is standard in classification tasks.
  • Task-Specific Bounds: Future work could establish more precise network width requirements that depend directly on the specific "bandwidth" or intrinsic complexity of the task at hand.

Summary updated

Abstract

We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that in the underparameterized regime the network learns eigenfunctions of an integral operator $T_K$ determined by the Neural Tangent Kernel at rates corresponding to their eigenvalues. For example, for uniformly distributed data on the sphere $S^{d - 1}$ and rotation invariant weight distributions, the eigenfunctions of $T_K$ are the spherical harmonics. Our results can be understood as describing a spectral bias in the underparameterized regime. The proofs use the concept of ``Damped Deviations'' where deviations of the NTK matter less for eigendirections with large eigenvalues. Aside from the underparameterized regime, the damped deviations point-of-view allows us to extend certain results in the literature in the overparameterized setting.
BibTeX
@inproceedings{bowman2022implicit,
  author = {Benjamin Bowman and Guido Montúfar},
  title = {{Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=a-1jPnzUIC},
}
Figure