Abstract
Modern deep learning models often generalize remarkably well despite being significantly overparameterized, a phenomenon that challenges classical learning theory. This paper presents a novel framework for analyzing generalization errors using functional calculus on the space of probability measures. By adopting a "mean-field" perspective for one-hidden layer neural networks, the authors establish sharp, non-asymptotic $O(1/n)$ convergence rates for generalization errors in KL-regularized learning problems. This approach provides a more precise quantification of model performance than traditional tools like VC-dimension or Rademacher complexity, which often become vacuous in the overparameterized regime.
Introduction
In classical learning theory, models with more parameters than data points are expected to overfit. However, modern neural networks defy this intuition, often generalizing well even when they perfectly interpolate training data. This discrepancy highlights the need for new theoretical tools to quantify the generalization error—the gap between a model's performance on training data versus unseen data.
The authors address this by focusing on the mean-field regime. In this setting, a one-hidden layer neural network is viewed not as a collection of individual weights, but as a distribution over the parameter space. This shift in perspective allows for the application of advanced mathematical techniques to find precise conditions that guarantee a fast decay of generalization error as the number of samples ($n$) increases.
The Mean-Field Perspective
The core of the analysis involves modeling neural networks as an expectation over a distribution of neurons. This allows the learning process to be treated as finding an optimal measure rather than a fixed vector of weights.
As shown in the figure above, the output $O$ of the network is represented as an average over $r$ hidden units, where each unit $h_j$ applies an activation function $\varphi$ to a weighted sum of inputs. In the mean-field limit, as $r$ becomes large, the empirical average converges to an integral over a probability measure, facilitating the use of functional derivatives.
Methodology
The paper's primary innovation is the use of differential calculus on the space of probability measures. The methodology consists of several key components:
- Functional Linear Derivatives: The authors utilize functional derivatives (specifically L-derivatives) to measure how the loss function and the learning algorithm react to small perturbations in the training data.
- KL-Regularized Empirical Risk Minimization (ERM): The training objective analyzed combines empirical risk with a Kullback-Leibler (KL) divergence term. This formulation leads to a "Gibbs measure" as the optimal solution, which is more amenable to stability analysis.
- Resampling Stability: By analyzing how the Gibbs measure changes when data points are resampled, the authors connect the algorithmic stability to generalization performance. If an algorithm's output is stable under minor data changes, it is more likely to generalize well.
Key Results
The framework yields several significant theoretical advancements:
- The $O(1/n)$ Rate: While many previous works found generalization rates of $O(1/\sqrt{n})$, this paper identifies conditions under which a much faster $O(1/n)$ rate is achieved. Crucially, these bounds are non-asymptotic, meaning they hold for any finite sample size $n$.
- Regularity over Smoothness: For neural networks, the $O(1/n)$ rate holds even for non-smooth activation functions like ReLU, provided they exhibit linear growth and the loss function is sufficiently regular.
- Unified Toolset: The authors derived new representations for both Weak Generalization Error (WGE) and $L_2$ Generalization Error (LGE) in terms of functional derivatives, providing a powerful new tool for the learning theory toolbox that can be applied to any algorithm represented as a map between probability measures.
Conclusion and Future Work
This research provides a rigorous mathematical foundation for understanding why overparameterized models generalize. By moving from weight-based analysis to measure-based analysis, the authors achieve tighter bounds and fewer restrictive assumptions than previous literature.
Future work aims to extend this functional calculus approach to deeper, multi-layer architectures. Additionally, while the current sharpest results rely on KL-regularization and specific moment assumptions for data distributions, future research may explore relaxing these requirements or applying the framework to optimization paths like Stochastic Gradient Descent (SGD) without explicit regularization.
Summary updated