Abstract
We consider the supervised learning problem with shallow neural networks. According to our unpublished experiments conducted several years prior to this study, we had noticed an interesting similarity between the distribution of hidden parameters after backprobagation (BP) training, and the ridgelet spectrum of the same dataset. Therefore, we conjectured that the distribution is expressed as a version of ridgelet transform, but it was not proven until this study. One difficulty is that both the local minimizers and the ridgelet transforms have an infinite number of varieties, and no relations are known between them. By using the integral representation, we reformulate the BP training as a strong-convex optimization problem and find a global minimizer. Finally, by developing ridgelet analysis on a reproducing kernel Hilbert space (RKHS), we write the minimizer explicitly and succeed to prove the conjecture. The modified ridgelet transform has an explicit expression that can be computed by numerical integration, which suggests that we can obtain the global minimizer of BP, without BP.
Introduction
Training neural networks typically relies on backpropagation (BP), an iterative process navigating a high-dimensional, non-convex loss landscape. A persistent concern in deep learning theory is the potential for these models to become trapped in suboptimal local minima. While empirical success suggests that global or near-global minima are often reached, a rigorous mathematical characterization of these optimal weight distributions has remained elusive.
This study bridges this gap by formalizing a "conjecture of similarity" observed in earlier experiments: the distribution of weights in a trained shallow network mirrors the spectrum of the ridgelet transform. By transitioning from a discrete sum of neurons to a continuous integral representation, we demonstrate that the global minimum is not only reachable but explicitly calculable.
Theoretical Framework: Convexification and Ridgelet Analysis
The core of our approach lies in reformulating the neural network's output. Instead of a finite sum of neurons, we represent the network as an integral over a coefficient function $\gamma(a, b)$ in a parameter space of weights $a$ and biases $b$. This shift to an integral representation transforms the training objective. While the standard discrete model is non-convex due to weights appearing inside the nonlinear activation function, the output of the integral model is linear with respect to the function $\gamma$.
Consequently, the training problem becomes a quadratic programming problem in an infinite-dimensional Hilbert space, which possesses a unique and reachable global minimum. To solve this optimization, we utilize ridgelet analysis. The ridgelet transform serves as a mathematical tool to decompose functions into "ridges." We prove that a modified version of this transform acts as a right inverse to the neural network operator. This allows us to derive Theorem 4.7, which provides an explicit expression for the optimal coefficient function that minimizes the risk.
Experimental Validation
To validate the theory, we analyzed 10 synthetic 1D datasets, including sinusoidal curves with noise, high-frequency oscillations, and Gaussian kernels. These datasets were designed to test how weight distributions reflect specific function properties like frequency and spatial location.
The results provide striking visual confirmation of our conjecture. When plotting the weights $(a, b)$ of networks trained via ADAM, the parameters cluster into distinct patterns—often a "star shape"—that align perfectly with the high-intensity regions of the ridgelet spectrum. This alignment suggests that standard optimization algorithms are implicitly converging toward the solution defined by the ridgelet transform.
Discussion and Key Insights
The implications of this work are twofold. First, it suggests the possibility of "training without training." For shallow networks, one could theoretically bypass iterative backpropagation entirely by performing direct numerical integration of the modified ridgelet transform.
Second, the study sheds light on implicit regularization. We observed that different optimizers (e.g., ADAM vs. BFGS) and activation functions (ReLU vs. Tanh) produce varying degrees of "sharpness" in the weight distributions. This indicates that common deep learning tools have inherent mathematical preferences for specific sparse solutions within the broader ridgelet spectrum.
Conclusion and Future Work
We have proven that the global minimum of a shallow neural network training problem is explicitly given by a modified ridgelet transform. While this result is currently limited to single-hidden-layer architectures, it provides a robust theoretical foundation for understanding the optimization landscape. Future research will focus on extending these integral representations to deep architectures and addressing the computational challenges of high-dimensional numerical integration.
Summary updated