Abstract
We establish a log-Sobolev inequality (LSI) for the stationary distribution of mean-field Langevin dynamics (MFLD) with a constant that is independent of the number of particles $N$. Our proof proceeds by establishing the existence of a Lipschitz transport map from the standard Gaussian measure via the reverse heat flow of Kim and Milman. This approach allows us to improve previous "doubly exponential" bounds to a "singly exponential" scaling, providing tighter guarantees for the convergence of these systems to equilibrium, particularly in the context of two-layer neural networks.
Introduction
Mean-Field Langevin Dynamics (MFLD) serves as a powerful framework for sampling from complex probability distributions by simulating a system of $N$ interacting particles. As $N$ approaches infinity, the system approximates a target "mean-field" distribution. A fundamental question in the analysis of these systems is the rate at which the $N$-particle system converges to its stationary state.
This convergence rate is typically governed by the log-Sobolev constant. Historically, a significant challenge has been that this constant often grows with $N$. If the constant scales poorly with the number of particles, the time required to reach equilibrium becomes prohibitive as the approximation becomes more refined. The goal of this research is to establish a "uniform-in-$N$" bound—a guarantee that the convergence rate remains stable regardless of the system size.
The Problem: Beyond Double Exponential Bounds
Previous theoretical attempts to bound the LSI constant for MFLD often resulted in "doubly exponential" scaling ($exp(exp(\dots))$). Such bounds are practically negligible for real-world parameters, as they suggest an astronomical time to convergence. Furthermore, these results often required highly restrictive conditions on the energy functionals involved.
To provide a meaningful guarantee for practitioners—especially those training two-layer neural networks in the mean-field regime—it is necessary to achieve a "singly exponential" scaling. This requires overcoming the recursive dependency where the convergence bound depends on the LSI constant, which in turn depends back on the convergence bound.
Methodology
Lipschitz Transport Map
Rather than analyzing the log-Sobolev inequality directly, we utilize a geometric approach. We prove the existence of a Lipschitz-continuous function that transforms a simple standard Gaussian measure into the complex target particle measure. A key property of LSI is that if such a map exists with a bounded Lipschitz constant, the LSI property is inherited from the Gaussian measure by the target measure.
Reverse Heat Flow and Tilt Stability
To construct this map, we employ the reverse heat flow method proposed by Kim and Milman. This involves analyzing how the measure evolves as it is smoothed by heat flow. We utilize a technical condition known as "tilt stability" to bound the covariance of the measure during this evolution.
Propagation of Chaos
To ensure the bounds remain independent of $N$, we leverage recent advancements in "propagation of chaos" results. Specifically, we build upon techniques that quantify how particles remain approximately independent over time. By using a refined argument (following Nitanda, 2024), we avoid the recursive dependency that previously led to double-exponential results.
Key Insights and Ideas
Linear vs. Displacement Convexity
While "displacement convexity" (convexity along Wasserstein geodesics) is a staple of optimal transport, this paper focuses on linear convexity (convexity along straight lines of probability measures). Linear convexity is the natural structure found in the loss surfaces of mean-field neural networks, making these results directly applicable to machine learning theory.
Dimension-Free Bounds
For the specific case of two-layer neural networks, we demonstrate that the Lipschitz constant—and consequently the convergence rate—can be made independent of the ambient dimension $d$. This holds provided the network architecture maintains certain smoothness properties, ensuring that the complexity of the input space does not degrade the sampling efficiency.
Main Results
- Theorem 2 (General Setting): We establish the existence of a Lipschitz transport map with a constant $L$ that scales as $exp(poly(\delta))$, where $\delta$ represents the strength of the interaction or perturbation. This confirms the sought-after singly exponential bound.
- Theorem 3 (Neural Network Setting): We refine this bound for two-layer networks, successfully removing the dependency on dimension $d$.
- Functional Inequalities: The establishment of a Lipschitz transport map provides more than just LSI; it automatically grants other functional tools, including Talagrand’s inequality and Gaussian isoperimetric inequalities for the particle system.
Conclusion and Future Work
This work provides a major theoretical milestone by stabilizing the LSI constant for mean-field systems. However, some limitations remain. Our results rely on the assumption that the mean-field energy functional is linearly convex, which may not cover non-convex landscapes where phase transitions occur. Additionally, the proof requires smooth energy functionals (bounded second and third variations), leaving the extension to non-smooth activations like ReLU as an open challenge. Future research may also explore the relationship between this transport map approach and concurrent work using Poincaré inequalities to reach even more general "flat-convex" settings.
Summary updated