Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum

Sho Sonoda, Isao Ishikawa, Masahiro Ikeda

Abstract

Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes.

Introduction

Despite the massive success of deep learning, characterizing the "landscape" of neural network parameters remains a challenge. Because neural networks are non-convex, finding and describing their local minima is notoriously difficult. Recent theory suggests that over-parametrization—using a massive number of neurons—simplifies this landscape, making global minima easier to find. However, most existing theories describe that a network converges, but not necessarily what it converges to. This paper provides an explicit closed-form expression for the global minimizers of ridge-regularized two-layer networks: the ridgelet spectrum.

The Ridgelet Transform and Integral Representation

The authors employ harmonic analysis, specifically the Ridgelet Transform, to represent functions using "ridges" (linear combinations found in neural network neurons). By treating the neural network as an integral over a continuous distribution of parameters rather than a sum of discrete neurons, they can leverage powerful mathematical tools.

A key innovation is the development of the ridgelet transform on the torus for periodic activation functions. This allows the use of Fourier analysis tools that are typically difficult to apply to standard unbounded activations like ReLU.

Convergence to the Ridgelet Spectrum

The core theoretical contribution is proving that the Finite Empirical Risk Minimizer (the weights found by training a finite network with $L_2$ regularization) converges to the continuous ridgelet spectrum as both the number of neurons and the number of samples go to infinity.

Specifically, the authors show that ridge regression acts as a "shrinkage estimator" on the ridgelet spectrum. The explicit formula for the optimal parameter distribution $\gamma^*$ is approximately $R[pf / (\beta + p)]$, where $R$ is the ridgelet transform, $f$ is the target function, $p$ is the data density, and $\beta$ is the regularization strength.

Empirical Evidence

To verify the theory, the authors trained 1,000 networks using Stochastic Gradient Descent (SGD) on synthetic 1D datasets. The resulting weight distributions show a remarkable visual resemblance to the heatmaps generated by the theoretical ridgelet spectrum.

Distribution of parameters trained by SGD for a ReLU network.

This visual confirmation holds across various activation functions, including periodic versions of ReLU, Tanh, and Gaussian functions, demonstrating the robustness of the ridgelet framework.

Insights and Limitations

This work suggests that neural networks are "structured approximators"—there is a principled mathematical reason why certain weight patterns emerge. It also provides a theoretical backbone for the "lazy regime," where weights barely move from their initial values if the initial distribution already covers the support of the ridgelet spectrum.

Currently, the proof is limited to two-layer networks and assumes periodic activation functions for mathematical tractability. Future work aims to extend these results to deep, multi-layer architectures and capture other types of implicit bias beyond $L_2$ regularization.

Summary updated

Abstract

Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes. We confirm the visual resemblance between the parameter distribution trained by SGD, and the ridgelet spectrum calculated by numerical integration through numerical experiments with finite models.
BibTeX
@inproceedings{sonoda2021ridge,
  author = {Sho Sonoda and Isao Ishikawa and Masahiro Ikeda},
  title = {{Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
}
Figure