Modern machine learning frequently encounters data that cannot be neatly packaged into finite-dimensional vectors. From continuous temperature curves in functional data analysis to complex structures mapped into Reproducing Kernel Hilbert Spaces (RKHS) for graph classification, infinite-dimensional data is increasingly common.
However, most existing neural network theory relies heavily on finite-dimensional assumptions. When naively applied to infinite dimensions, standard optimization guarantees often blow up due to the "curse of dimensionality." Furthermore, while the Neural Tangent Kernel (NTK) regime offers a mathematically convenient way to analyze networks, it fails to capture feature learning—the crucial ability of a neural network to adapt its internal representations to the data.
This paper bridges a major gap in deep learning theory by providing the first quantitative global optimization guarantees for two-layer neural networks trained on infinite-dimensional data within the mean-field regime, a setting that explicitly allows for feature learning.
The Challenge of Infinite Dimensions
When dealing with infinite-dimensional inputs, the parameters of the network's neurons must also reside in an infinite-dimensional space (specifically, a separable Hilbert space). The core challenge is that traditional convergence bounds depend on the dimension of the parameter space. If the dimension is infinite, these bounds become meaningless.
To overcome this, the authors shift the perspective: instead of tracking individual weights, they model the neural network as an average over $M$ neurons and focus on the distribution of these parameters. By leveraging mathematical abstractions like Gaussian measures and Radon-Nikodym derivatives, they successfully derive convergence rates that are entirely independent of the input dimension.
Algorithmic Innovations
To optimize this infinite-dimensional parameter distribution, the authors extend existing particle optimization frameworks, introducing three key algorithmic contributions:
- Infinite-Dimensional Particle Dual Averaging (PDA): An extension of Nesterov’s dual averaging. It employs a double-loop structure where the inner loop samples new "particles" (neurons) to represent the evolving parameter distribution.
- Infinite-Dimensional Particle Stochastic Dual Coordinate Ascent (P-SDCA): Tailored for finite-sum objectives (such as standard training datasets), this method accelerates convergence significantly.
- Infinite-Dimensional Langevin Dynamics: To sample particles in the PDA inner loop, the authors adapt gradient Langevin dynamics to Hilbert spaces. They utilize a semi-implicit Euler scheme and Galerkin approximation to ensure computations remain numerically stable and finite while theoretically representing an infinite-dimensional process.
Key Theoretical and Empirical Results
The paper establishes rigorous theoretical guarantees and validates them through a controlled "Teacher-Student" setup for non-linear functional regression, where inputs are square-integrable functions ($L^2([0, 1])$) and Sobolev spaces ($H^1([0, 1])$).
- Generalization Guarantee: The authors prove a generalization error bound of $O(1/\sqrt{n})$, where $n$ is the number of training samples. This demonstrates that the model avoids overfitting, even when utilizing infinite-dimensional neurons.
- Provable Optimization Rates: The PDA algorithm achieves a polynomial convergence rate ($O(1/T)$) in the online setting. Remarkably, the P-SDCA algorithm achieves exponential convergence in the finite-sample setting.
- The Advantage of Feature Learning: In empirical tests, the mean-field neural networks achieved significantly lower test losses than classical "non-adaptive" baselines like Ridge Regression and the Nadaraya-Watson estimator. While linear or kernel-based methods rely on fixed basis functions, mean-field networks adapt their bases (the neurons) to the specific structure of the functional data.
Limitations and Future Directions
While a significant theoretical milestone, the framework has a few limitations that pave the way for future research:
- Regularization Sensitivity: The convergence bounds exhibit an exponential dependency on the regularization parameter ($\lambda_2$). This is a known bottleneck in Langevin-based sampling methods, meaning that very small regularization can drastically slow down the algorithm.
- Architectural Limits: The current theory is restricted to two-layer (one hidden layer) networks. Extending these global optimization guarantees to deep, multi-layer architectures in the mean-field regime remains a formidable open challenge.
- Computational Overhead: The double-loop structure (outer optimization paired with inner Langevin sampling) is computationally more intensive than standard Stochastic Gradient Descent (SGD), potentially limiting its immediate application on extremely large-scale datasets without further optimization.
Ultimately, this research provides a rigorous mathematical foundation for understanding why and how neural networks can effectively learn features from infinitely complex data, opening new doors for functional data analysis and advanced kernel methods.
Summary updated