Abstract: We study the approximation gap between the dynamics of a polynomial-width neural network and its infinite-width counterpart, both trained using projected gradient descent in the mean-field scaling regime. We demonstrate how to tightly bound this approximation gap through a differential equation governed by the mean-field dynamics. A key factor influencing the growth of this ODE is the local Hessian of each particle, defined as the derivative of the particle's velocity in the mean-field dynamics with respect to its position. We apply our results to the canonical feature learning problem of estimating a well-specified single-index model; we permit the information exponent to be arbitrarily large, leading to convergence times that grow polynomially in the ambient dimension d. We show that, due to a certain "self-concordance" property in these problems -- where the local Hessian of a particle is bounded by a constant times the particle's velocity -- polynomially many neurons are sufficient to closely approximate the mean-field dynamics throughout training.
The Challenge of Long-Time Training
In the study of neural networks, the Mean-Field regime provides a powerful framework for understanding how neurons adapt to data during training. However, a significant theoretical gap exists: how closely does a finite-width network with $m$ neurons track its idealized infinite-width counterpart? This is known as the Propagation of Chaos problem.
Standard approaches rely on Gronwall-type estimates, which predict that the approximation error grows exponentially over time. This restricts our theoretical guarantees to very short durations—typically $O(\log d)$ or $O(1)$. Yet, many practical feature-learning tasks, such as learning Single-Index Models (SIM), require training times that scale polynomially with the dimension $d$. This paper breaks the logarithmic barrier, proving that finite networks can track mean-field dynamics for $T = \text{poly}(d)$ time.
Methodology: Potential-Based Analysis
The authors move beyond global worst-case error bounds, introducing a potential-based analysis that accounts for the specific geometry of the loss landscape. They decompose the approximation gap into several manageable terms, accounting for discretization, coupling, and interaction effects.
The core of this approach involves:
- Local Hessian Analysis: Defining a "local Hessian" for each neuron to describe how its velocity changes relative to its position.
- Self-Concordance: Identifying a property where the local Hessian is bounded by the neuron's own velocity. This ensures that when a neuron is moving slowly (e.g., near a saddle point), the forces pushing it off-track are also weak.
- Interaction Stability: Showing that neuron-to-neuron interactions, often viewed as noise, actually provide a stabilizing "repulsion" effect because the interaction Hessian is Positive Semi-Definite (PSD).
Key Insights and Results
The most important realization is the "Saddle Protection": neurons are most vulnerable to diverging from the mean-field limit when they are near saddle points. However, because their velocity is low there, the self-concordance property ensures the error doesn't grow exponentially during this phase.
Primary Results: 1. Polynomial Time Guarantee: For a network with $m = \text{poly}(d)$ neurons, the finite-width dynamics closely approximate the mean-field limit for $T = \text{poly}(d)$. 2. Single-Index Model Convergence: Applied to SIMs with high information exponents ($k^* \ge 4$), finite networks achieve small population loss within the same timeframe as the infinite-width limit. 3. Empirical Validation: Simulations on SIM, XOR4, and Staircase architectures confirm that function error decays at a rate of $1/m$ even over long durations, matching theoretical predictions.
Limitations and Future Work
While the results are robust, the current proof for Local Strong Convexity (LSC) assumes target neurons are discrete "atomic" points. While simulations on continuous "circle" targets look promising, the formal proof for non-atomic targets remains open. Additionally, the analysis relies on certain rotational symmetries and requires the landscape to be locally well-behaved near the end of training. Extending this to more rugged or non-convex landscapes is a key direction for future research.
Summary updated