When we train a standard neural network, we track the updates of individual weights. From a mathematical perspective, this optimization landscape is notoriously non-convex, filled with local minima and saddle points that should, in theory, trap our algorithms. Yet, in practice, overparameterized networks consistently find global minima.
To understand why, a powerful research direction shifts our perspective from individual parameters to the collective behavior of the entire network. By treating the neurons as a continuous fluid rather than discrete particles, we enter the mean-field regime. In this infinite-width limit, the evolution of a neural network can be modeled as a Wasserstein gradient flow—a trajectory that transforms a highly non-convex optimization problem into a provably convergent, convex evolution over probability measures.
The Particle to Fluid Duality
To build intuition, imagine a two-layer neural network as a collection of discrete particles scattered across a landscape. Each particle represents the weights of a single hidden neuron. As gradient descent progresses, these particles move along the loss landscape to minimize the network's error.
When the network is narrow, tracking these individual particles is chaotic. However, as the number of hidden units approaches infinity, the discrete particles blur into a continuous population density. Instead of tracking a massive vector of individual weights, we can track how the overall probability distribution of the weights, $\rho(w, t)$, evolves over time.
This transition from discrete updates to continuous distributions is known as the mean-field limit. As shown by Mei Song et al. in Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit and independently by Eric Vanden‐Eijnden in Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach, the collective behavior of these neurons is governed by a partial differential equation known as the continuity equation. Francis Williams et al. describe this in Gradient Dynamics of Shallow Univariate ReLU Networks as an Eulerian perspective: rather than following individual neurons, we observe how the entire cloud of neurons collectively flows toward configurations that best fit the data.
Why it Works: Convexity in Wasserstein Space
The true magic of the mean-field perspective lies in how it reshapes our optimization geometry. If you optimize a network in its standard parameter space $\mathbb{R}^d$, the objective function is highly non-convex. But when you lift the problem into the space of probability distributions $\mathcal{P}(\mathbb{R}^d)$, the objective function becomes a convex functional of the distribution $\rho$.
Because the network's output is an average over its neurons, the prediction depends linearly on the density $\rho$. Consequently, standard convex loss functions (like square loss or logistic loss) remain convex with respect to $\rho$. As Eric Vanden‐Eijnden demonstrates, this structural convexity guarantees that the continuous fluid flow will reliably converge toward a global optimum, bypassing the local traps of the finite-dimensional landscape.
To navigate this space of distributions, the natural geometry is dictated by optimal transport—specifically, the 2-Wasserstein metric. The Wasserstein metric defines the distance between two distributions based on the minimum "work" required to transport mass from one configuration to another. A Wasserstein gradient flow is simply the path of steepest descent that the weight distribution takes through this geometry.
Using this framework, Lénaïc Chizat et al. in Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss characterized the implicit bias of wide networks, proving that the distribution of neurons naturally deforms to find max-margin solutions. Taking this geometric perspective to its logical extreme, Sergei Gukov et al. noted in Rigor with Machine Learning from Field Theory to the Poincaré Conjecture that under specific infinite-width conditions, these optimization trajectories can be formally equated to geometric evolutions like the Ricci flow found in theoretical physics.
Injecting Noise: Mean-Field Langevin Dynamics
While the deterministic mean-field limit is theoretically powerful, pure gradient descent can still stall if the initial distribution lacks support over critical regions of the parameter space. To ensure thorough exploration and faster convergence, researchers introduce noise into the training process, leading to Mean-Field Langevin Dynamics (MFLD).
By adding Gaussian noise to the gradient updates of individual particles, the continuous distribution undergoes a combination of gradient-driven drift and thermal diffusion. Mathematically, adding this diffusion term transforms the Wasserstein gradient flow of the loss into a gradient flow of the Kullback-Leibler (KL) divergence relative to a target Gibbs distribution.
Deterministic Flow: ∂ρ/∂t = ∇ · (ρ ∇ (Loss)) [Drift toward minima]
Noisy Flow (MFLD): ∂ρ/∂t = ∇ · (ρ ∇ (Loss)) + Δρ [Drift + Diffusion/Exploration]
In Convex Analysis of the Mean Field Langevin Dynamics, Atsushi Nitanda et al. showed that this entropic regularization smooths out the optimization landscape, allowing the distribution to converge linearly to a unique global optimum. Lénaïc Chizat further established in Mean-Field Langevin Dynamics : Exponential Convergence and Annealing that under certain functional inequalities, this noisy measure-valued flow achieves exponential convergence rates.
Crucially, MFLD enables true feature learning. Unlike the static "lazy" or kernel regime where neurons barely move from their initializations, MFLD actively reshapes the underlying distribution to discover hidden structures in data. For instance, Taiji Suzuki et al. in Feature learning via mean-field Langevin dynamics: classifying sparse parities and beyond and Atsushi Nitanda et al. in Improved statistical and computational complexity of the mean-field Langevin dynamics under structured data proved that MFLD can efficiently learn complex "sparse parity" functions and exploit anisotropic data structures, bypassing the sample complexity bottlenecks that cripple standard kernel methods. Similarly, Alireza Mousavi Hosseini et al. demonstrated in Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics that constraining these noisy flows to a hypersphere can unlock polynomial-time training for low-dimensional multi-index structures.
Broadening the Horizon
The framework of modeling optimization as a distribution-dependent flow is not restricted to shallow, two-layer networks. It provides a unified mathematical language across several domains:
- Stochastic Gradient Descent: To bridge the gap between discrete, mini-batch updates and continuous fluid limits, Benjamin Gess et al. developed a model called Stochastic Modified Flows in Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent, capturing how parameter evolution depends on the overall population density even under small learning rates.
- Transformers: As models scale to billions of parameters, Cheng Gao et al. extended this framework to massive architectures in Global Convergence in Training Large-Scale Transformers, proving that the training of high-dimensional Transformers can be modeled as a predictable fluid flow that reliably converges to global minima.
- Reinforcement Learning: In policy optimization, non-convexity is a persistent challenge. By framing the evolution of single-hidden-layer neural network policies as a Wasserstein gradient flow, Andrea Agazzi et al. in Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime and James-Michael Leahy et al. in Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime proved that entropy-regularized policy gradients in the mean-field regime guarantee global convergence to the optimal policy.
- Deep Ensembles: Instead of viewing a single network as a collection of neurons, one can view an ensemble of distinct models as particles in a shared parameter space. Veit David Wild et al. utilized Wasserstein gradient flows in A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods to treat ensembles as interacting particle systems moving toward an optimal posterior distribution, providing a formal bridge between deep ensembles and variational inference.
By shifting our focus from the position of individual parameters to the continuous deformation of their probability distributions, Wasserstein gradient flows explain the remarkable success of modern deep learning. They reveal that underneath the chaotic, non-convex surface of neural network training lies a structured, predictable, and globally convergent fluid flow.
Key Foundational Work
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit (2019) — contributed 1 concept elements to this direction.
Gradient Dynamics of Shallow Univariate ReLU Networks (2019) — contributed 1 concept elements to this direction.
A mean field view of the landscape of two-layer neural networks (2018) — contributed 1 concept elements to this direction.