Ridge Regression with Over-parametrized Two-Layer Networks Converge to Ridgelet Spectrum

Sho Sonoda, Isao Ishikawa, Masahiro Ikeda

Introduction

In modern deep learning, over-parametrized models—those with significantly more parameters than training samples—are the norm. Contrary to classical statistical intuition, these models often generalize exceptionally well. A persistent challenge in the field is understanding the nature of the "good" local minima these networks find. While theories such as the Neural Tangent Kernel (NTK) and Mean-Field theory establish convergence to global minima, they frequently lack an explicit, closed-form description of the final parameter state.

This paper bridges this gap by providing a precise characterization of the parameter distribution using harmonic analysis, specifically through the lens of the ridgelet transform.

Methodology: The Ridgelet Transform

The authors employ the Ridgelet Transform, a mathematical tool analogous to the Fourier transform. While the Fourier transform decomposes signals into sine waves, the ridgelet transform decomposes functions into "ridges" of the form $\sigma(a \cdot x - b)$. This form is mathematically identical to the operation of a single neuron in a two-layer network.

To ensure mathematical rigor and satisfy the Plancherel formula, the authors developed a theory of ridgelet transforms on a torus ($\mathbb{T}$), enabling the use of periodic activation functions. By treating the network as an integral over a continuous parameter space and focusing on Regularized Empirical Risk Minimization (RERM) with ridge regression ($L^2$ regularization), they identify a unique solution among the manifold of global minima.

Key Insights

Parameters as a Spectrum

The central finding is that the weights of a trained network are not merely arbitrary values; they constitute a "ridgelet spectrum." In this view, a trained network is a physical realization of the ridgelet transform of the target function.

The Shrinkage Effect

The authors derive an explicit formula for the optimal parameter distribution $\gamma^*$: $$\gamma^* \approx R\left[\frac{pf}{\beta + p}\right]$$ where $R$ is the ridgelet transform, $f$ is the target function, $p$ is the data density, and $\beta$ is the regularization parameter. This reveals that $L^2$ regularization acts as a shrinkage estimator, dampening the influence of the target function in regions where data is sparse or noise is prevalent.

Explaining "Lazy Training"

These results provide a theoretical foundation for the "lazy regime." If the initial random weights already span the support of the ridgelet spectrum, the training process (e.g., SGD) primarily adjusts coefficients rather than moving weights extensively to achieve a solution.

Results and Numerical Verification

The paper mathematically proves that as the number of neurons and samples approach infinity, the parameter distribution of a finite network converges to the continuous ridgelet spectrum.

Numerical experiments on synthetic 1D functions, such as $f(x) = \sin(2\pi x)$, confirm this convergence. Scatter plots of weights from networks trained via Stochastic Gradient Descent (SGD) show a striking alignment with the heatmaps of the theoretically calculated ridgelet spectrum.

Comparison of SGD-trained parameters (top) and the theoretical ridgelet spectrum (bottom) for Gaussian, Tanh, and ReLU activations.

As illustrated in the figure, the empirical parameter distributions (top row) exhibit the same patterns as the theoretical spectrum (bottom row) across various activation functions. The $L^2$ penalty effectively regularizes the null space, leading to a stable and unique global minimizer.

Conclusion

This research offers a rare, transparent look into the internal state of over-parametrized networks. While currently limited to two-layer architectures and periodic activations, it sets the stage for future work extending these principles to deeper networks and more general regularization techniques. Understanding the ridgelet spectrum provides a powerful tool for demystifying the success of over-parametrized models in modern AI.

Summary updated

Abstract

Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes. We confirm the visual resemblance between the parameter distribution trained by SGD, and the ridgelet spectrum calculated by numerical integration through numerical experiments with finite models.
BibTeX
@inproceedings{sonoda2021ridge,
  author = {Sho Sonoda and Isao Ishikawa and Masahiro Ikeda},
  title = {{Ridge Regression with Over-parametrized Two-Layer Networks Converge to Ridgelet Spectrum}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
}
Figure