Abstract
This research provides a rigorous theoretical foundation for employing shallow neural networks as a regularization mechanism for ill-posed inverse problems. By treating the number of neurons (network width) as a formal regularization parameter, the authors prove that these architectures can stably reconstruct signals from noisy measurements. The study establishes a quantitative link between noise levels and optimal network width, demonstrating that shallow networks in Barron spaces can effectively bypass the curse of dimensionality.
The Challenge of Inverse Problems
Inverse problems are ubiquitous in scientific imaging, from medical CT scans to geophysical surveys. The goal is to reconstruct an internal state from indirect, noisy observations. These problems are inherently "ill-posed," meaning that minor measurement noise can lead to disproportionately large errors in the reconstruction. Traditional mathematical approaches rely on regularization—incorporating prior knowledge to stabilize the solution. While deep learning has seen empirical success in this domain, a rigorous theoretical "guarantee" for neural network-based regularization has been largely missing.
Methodology: Shallow Networks and Barron Spaces
The authors focus on two-layer (shallow) neural networks operating within Barron spaces. These functional spaces are particularly suited for shallow networks because they allow for efficient approximation without the exponential dependence on input dimension typically found in classical Sobolev or Hölder spaces.
Two primary regularization strategies are proposed:
- Expanding Neural Network Method: An iterative approach where neurons are added incrementally until the residual error matches the expected noise level (the Morozov discrepancy principle). Here, the network width $n$ serves as the regularization parameter.
- Neural Network-based Tikhonov Regularization: A variational method that adds a penalty term based on the Barron norm to the loss function, directly controlling the stability of the reconstruction.
Key Theoretical Insights
The core contribution of this work is the proof that network width $n$ acts as a regularization parameter. In the context of ill-posed problems, a network that is too wide risks "overfitting the noise," while one that is too narrow cannot capture the underlying signal.
The authors derive a scaling law relating the sufficient network size $n(\delta)$ to the noise level $\delta$: $$n(\delta) = O(\delta^{-2/\theta})$$ where $\theta$ represents the smoothness of the forward operator. This quantitative link ensures that as noise decreases, the network can grow to provide a more accurate reconstruction without losing stability. Furthermore, by utilizing Barron spaces, the approximation error remains $O(1/\sqrt{n})$, effectively bypassing the "curse of dimensionality" that plagues many other approximation schemes, as illustrated in the comparative literature summary above.
Experimental Validation
The theoretical framework was validated using three benchmark inverse problems:
- 1D Fredholm Integral Equation: A linear problem common in physical sensing.
- 1D Auto-convolution Volterra Equation: A nonlinear problem testing the limits of injectivity.
- 2D Electrical Impedance Tomography (EIT): A highly nonlinear and difficult medical imaging problem.
In these tests, the "Expanding Neural Network" method achieved higher accuracy with fewer neurons than standard Tikhonov approaches. The error typically followed a U-shaped curve: decreasing as neurons were added to improve representation, then increasing as excessive width began to amplify noise.
Conclusion and Future Directions
This work bridges the gap between empirical deep learning and classical regularization theory. While currently limited to shallow networks, it provides a blueprint for analyzing more complex architectures. Future research will aim to extend these proofs to deep neural networks (DNNs) and graph-based architectures, as well as addressing the non-convex nature of the underlying optimization landscape.
Summary updated