Abstract
Characterizing the inductive bias of neural networks remains a central challenge in deep learning theory. This paper provides a mathematical bridge between neural network parameters and harmonic analysis, proving that in over-parametrized two-layer networks, the distribution of learned weights converges to a specific closed-form expression: the ridgelet spectrum of the target function. This result offers a precise characterization of how $L^2$ regularization shapes the final model, providing a theoretical foundation for understanding the "local minima" favored by stochastic gradient descent (SGD).
Introduction: The Mystery of Over-Parametrization
Despite the empirical success of deep learning, a fundamental question persists: why do neural networks generalize well even when they have far more parameters than data points? In this "over-parametrized" regime, there are infinitely many parameter configurations that can perfectly fit the training data. Yet, training processes like SGD consistently favor solutions that generalize.
Previous theoretical frameworks, such as the Neural Tangent Kernel (NTK) or Mean-Field theory, describe the dynamics of convergence but often lack an explicit, closed-form description of the final parameter distribution. This research aims to fill that gap by using the tools of integral transforms to provide an explicit expression for the learned weights.
Theoretical Framework: Ridgelets on the Torus
The authors leverage the ridgelet transform, an integral transform designed to represent functions as sums of "ridges"—mathematical structures that mirror the hidden units in a neural network.
Methodological Contributions
- Ridgelets on the Torus: Traditional ridgelet theory often struggles with non-integrable activation functions like ReLU. To overcome this, the authors developed a new theory of the ridgelet transform on the torus ($ℝ$). This approach allows for the analysis of periodic versions of common activations (ReLU, Tanh, Gaussian) using Fourier analysis.
- Integral Representation: Rather than treating weights as discrete vectors, the network is modeled as a continuous integral. The authors then analyze the Regularized Empirical Risk Minimization (RERM) problem with an $L^2$ (ridge) penalty.
- Mathematical Proofs: The core contribution is a proof showing that as the number of parameters and samples approach infinity, the unique minimizer of the risk function is a "shrunk" version of the ridgelet transform of the target function.
The Ridgelet Inductive Bias
The paper's central insight is that learned parameters are not random; they directly reflect the target function's ridgelet spectrum. Specifically, the optimal parameter distribution $\gamma^*$ is approximately $R[pf / (\beta + p)]$, where $R$ is the ridgelet transform, $f$ is the target, $p$ is the data density, and $\beta$ is the regularization strength.
This perspective reveals that the $L^2$ penalty (weight decay) acts as a "filter," selecting the unique ridgelet solution from the infinite possibilities in the network's null space. Furthermore, it provides a theoretical explanation for "lazy learning": if initial random weights already cover the support of the ridgelet spectrum, the network primarily needs to adjust output coefficients rather than moving weights significantly.
Results and Visual Validation
To validate their theoretical findings, the authors conducted experiments using synthetic 1D datasets, including simple sine waves, translated Gaussians, and square waves. These datasets allow for clear visualization of parameter distributions in a 2D space defined by position parameters $a$ and $b$.
The empirical results show a striking visual resemblance between the weight distributions of networks trained via SGD and the mathematically calculated ridgelet spectra.
As shown in the figure above, the geometric structures formed by the parameters are accurately predicted by the ridgelet theory. This holds true across different activation functions, provided they meet a specific "admissibility condition."
Conclusion and Future Directions
This work provides a rigorous characterization of the global minimizer in over-parametrized two-layer networks. While it establishes a powerful link between deep learning and harmonic analysis, several avenues for future research remain:
- Deep Networks: The current proof is limited to two-layer architectures. Extending this to deeper networks, perhaps as a sequence of ridgelet transforms, is a significant open challenge.
- SGD Trajectory: While experiments show SGD reaches the predicted state, a formal proof of the implicit bias of the SGD trajectory itself (as opposed to the final objective) is still needed.
- Non-Periodic Activations: The use of periodic constraints on the torus was a mathematical necessity for this proof. Extending these results to standard activations on the entire real line while maintaining energy conservation (Plancherel formula) remains a goal for future theoretical work.
Summary updated