Introduction
In modern deep learning, over-parametrized models—those with significantly more parameters than training samples—are the norm. Contrary to classical statistical intuition, these models often generalize exceptionally well. A persistent challenge in the field is understanding the nature of the "good" local minima these networks find. While theories such as the Neural Tangent Kernel (NTK) and Mean-Field theory establish convergence to global minima, they frequently lack an explicit, closed-form description of the final parameter state.
This paper bridges this gap by providing a precise characterization of the parameter distribution using harmonic analysis, specifically through the lens of the ridgelet transform.
Methodology: The Ridgelet Transform
The authors employ the Ridgelet Transform, a mathematical tool analogous to the Fourier transform. While the Fourier transform decomposes signals into sine waves, the ridgelet transform decomposes functions into "ridges" of the form $\sigma(a \cdot x - b)$. This form is mathematically identical to the operation of a single neuron in a two-layer network.
To ensure mathematical rigor and satisfy the Plancherel formula, the authors developed a theory of ridgelet transforms on a torus ($\mathbb{T}$), enabling the use of periodic activation functions. By treating the network as an integral over a continuous parameter space and focusing on Regularized Empirical Risk Minimization (RERM) with ridge regression ($L^2$ regularization), they identify a unique solution among the manifold of global minima.
Key Insights
Parameters as a Spectrum
The central finding is that the weights of a trained network are not merely arbitrary values; they constitute a "ridgelet spectrum." In this view, a trained network is a physical realization of the ridgelet transform of the target function.
The Shrinkage Effect
The authors derive an explicit formula for the optimal parameter distribution $\gamma^*$: $$\gamma^* \approx R\left[\frac{pf}{\beta + p}\right]$$ where $R$ is the ridgelet transform, $f$ is the target function, $p$ is the data density, and $\beta$ is the regularization parameter. This reveals that $L^2$ regularization acts as a shrinkage estimator, dampening the influence of the target function in regions where data is sparse or noise is prevalent.
Explaining "Lazy Training"
These results provide a theoretical foundation for the "lazy regime." If the initial random weights already span the support of the ridgelet spectrum, the training process (e.g., SGD) primarily adjusts coefficients rather than moving weights extensively to achieve a solution.
Results and Numerical Verification
The paper mathematically proves that as the number of neurons and samples approach infinity, the parameter distribution of a finite network converges to the continuous ridgelet spectrum.
Numerical experiments on synthetic 1D functions, such as $f(x) = \sin(2\pi x)$, confirm this convergence. Scatter plots of weights from networks trained via Stochastic Gradient Descent (SGD) show a striking alignment with the heatmaps of the theoretically calculated ridgelet spectrum.
As illustrated in the figure, the empirical parameter distributions (top row) exhibit the same patterns as the theoretical spectrum (bottom row) across various activation functions. The $L^2$ penalty effectively regularizes the null space, leading to a stable and unique global minimizer.
Conclusion
This research offers a rare, transparent look into the internal state of over-parametrized networks. While currently limited to two-layer architectures and periodic activations, it sets the stage for future work extending these principles to deeper networks and more general regularization techniques. Understanding the ridgelet spectrum provides a powerful tool for demystifying the success of over-parametrized models in modern AI.
Summary updated