Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime

Naoki Nishikawa, Taiji Suzuki, Atsushi Nitanda, Denny Wu

Modern machine learning frequently encounters data that cannot be neatly packaged into finite-dimensional vectors. From continuous temperature curves in functional data analysis to complex structures mapped into Reproducing Kernel Hilbert Spaces (RKHS) for graph classification, infinite-dimensional data is increasingly common.

However, most existing neural network theory relies heavily on finite-dimensional assumptions. When naively applied to infinite dimensions, standard optimization guarantees often blow up due to the "curse of dimensionality." Furthermore, while the Neural Tangent Kernel (NTK) regime offers a mathematically convenient way to analyze networks, it fails to capture feature learning—the crucial ability of a neural network to adapt its internal representations to the data.

This paper bridges a major gap in deep learning theory by providing the first quantitative global optimization guarantees for two-layer neural networks trained on infinite-dimensional data within the mean-field regime, a setting that explicitly allows for feature learning.

The Challenge of Infinite Dimensions

When dealing with infinite-dimensional inputs, the parameters of the network's neurons must also reside in an infinite-dimensional space (specifically, a separable Hilbert space). The core challenge is that traditional convergence bounds depend on the dimension of the parameter space. If the dimension is infinite, these bounds become meaningless.

To overcome this, the authors shift the perspective: instead of tracking individual weights, they model the neural network as an average over $M$ neurons and focus on the distribution of these parameters. By leveraging mathematical abstractions like Gaussian measures and Radon-Nikodym derivatives, they successfully derive convergence rates that are entirely independent of the input dimension.

Algorithmic Innovations

To optimize this infinite-dimensional parameter distribution, the authors extend existing particle optimization frameworks, introducing three key algorithmic contributions:

  1. Infinite-Dimensional Particle Dual Averaging (PDA): An extension of Nesterov’s dual averaging. It employs a double-loop structure where the inner loop samples new "particles" (neurons) to represent the evolving parameter distribution.
  2. Infinite-Dimensional Particle Stochastic Dual Coordinate Ascent (P-SDCA): Tailored for finite-sum objectives (such as standard training datasets), this method accelerates convergence significantly.
  3. Infinite-Dimensional Langevin Dynamics: To sample particles in the PDA inner loop, the authors adapt gradient Langevin dynamics to Hilbert spaces. They utilize a semi-implicit Euler scheme and Galerkin approximation to ensure computations remain numerically stable and finite while theoretically representing an infinite-dimensional process.

Key Theoretical and Empirical Results

The paper establishes rigorous theoretical guarantees and validates them through a controlled "Teacher-Student" setup for non-linear functional regression, where inputs are square-integrable functions ($L^2([0, 1])$) and Sobolev spaces ($H^1([0, 1])$).

  • Generalization Guarantee: The authors prove a generalization error bound of $O(1/\sqrt{n})$, where $n$ is the number of training samples. This demonstrates that the model avoids overfitting, even when utilizing infinite-dimensional neurons.
  • Provable Optimization Rates: The PDA algorithm achieves a polynomial convergence rate ($O(1/T)$) in the online setting. Remarkably, the P-SDCA algorithm achieves exponential convergence in the finite-sample setting.
  • The Advantage of Feature Learning: In empirical tests, the mean-field neural networks achieved significantly lower test losses than classical "non-adaptive" baselines like Ridge Regression and the Nadaraya-Watson estimator. While linear or kernel-based methods rely on fixed basis functions, mean-field networks adapt their bases (the neurons) to the specific structure of the functional data.

Limitations and Future Directions

While a significant theoretical milestone, the framework has a few limitations that pave the way for future research:

  • Regularization Sensitivity: The convergence bounds exhibit an exponential dependency on the regularization parameter ($\lambda_2$). This is a known bottleneck in Langevin-based sampling methods, meaning that very small regularization can drastically slow down the algorithm.
  • Architectural Limits: The current theory is restricted to two-layer (one hidden layer) networks. Extending these global optimization guarantees to deep, multi-layer architectures in the mean-field regime remains a formidable open challenge.
  • Computational Overhead: The double-loop structure (outer optimization paired with inner Langevin sampling) is computationally more intensive than standard Stochastic Gradient Descent (SGD), potentially limiting its immediate application on extremely large-scale datasets without further optimization.

Ultimately, this research provides a rigorous mathematical foundation for understanding why and how neural networks can effectively learn features from infinitely complex data, opening new doors for functional data analysis and advanced kernel methods.

Summary updated

Abstract

Analysis of neural network optimization in the mean-field regime is important as the setting allows for feature learning. Existing theory has been developed mainly for neural networks in finite dimensions, i.e., each neuron has a finite-dimensional parameter. However, the setting of infinite-dimensional input naturally arises in machine learning problems such as nonparametric functional data analysis and graph classification. In this paper, we develop a new mean-field analysis of two-layer neural network in an infinite-dimensional parameter space. We first give a generalization error bound, which shows that the regularized empirical risk minimizer properly generalizes when the data size is sufficiently large, despite the neurons being infinite-dimensional. Next, we present two gradient-based optimization algorithms for infinite-dimensional mean-field networks, by extending the recently developed particle optimization framework to the infinite-dimensional setting. We show that the proposed algorithms converge to the (regularized) global optimal solution, and moreover, their rates of convergence are of polynomial order in the online setting and exponential order in the finite sample setting, respectively. To our knowledge this is the first quantitative global optimization guarantee of neural network on infinite-dimensional input and in the presence of feature learning.
BibTeX
@inproceedings{nishikawa2022two,
  author = {Naoki Nishikawa and Taiji Suzuki and Atsushi Nitanda and Denny Wu},
  title = {{Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=RATusRo9xZ},
}
Figure