In traditional machine learning, we view a neural network as a collection of discrete weights—individual "particles" that we nudge around a high-dimensional landscape to minimize a loss function. However, as networks grow wider, this particle-based view becomes mathematically cumbersome. A more powerful perspective, championed by researchers like Weinan E, Chao Ma, and Lei Wu in Machine Learning from a Continuous Viewpoint, treats the network as a continuous distribution of neurons.
By shifting from "weights as points" to "weights as a density," the chaotic, non-convex optimization of deep learning reveals a hidden, elegant structure: a Wasserstein gradient flow.
From Particles to Fluid Dynamics
When a neural network is infinitely wide, we no longer care about the specific value of the 42nd neuron. Instead, we care about the distribution of all neurons. This is the "mean-field" regime. As Song Mei, Andrea Montanari, and colleagues demonstrate in A Mean Field View of the Landscape of Two-Layers Neural Networks, the training of these networks can be modeled as the evolution of a probability measure.
Imagine the neurons as a cloud of gas. Gradient descent isn't just moving points; it is a force field that deforms and shifts this cloud. The math that describes this "mass-moving" process is Optimal Transport. Specifically, the optimization follows a gradient flow in the Wasserstein space—a geometric space where the distance between two distributions is the minimum "work" required to transform one into the other.
Why Global Convergence is Possible
The most striking result of this viewpoint is that it explains why over-parameterized networks find global optima. While the loss landscape is notoriously non-convex with respect to individual weights, it often becomes convex (or "displacement convex") with respect to the distribution of those weights.
Lénaïc Chizat and Francis Bach show in On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport that in the many-particle limit, the gradient flow in Wasserstein space reliably converges to a global minimizer. By "lifting" the problem into the infinite-dimensional space of measures, the traps and local minima of the finite-dimensional landscape effectively disappear.
Feature Learning vs Lazy Training
This mean-field perspective is distinct from the "Neural Tangent Kernel" (NTK) or "lazy training" regime. In the NTK regime, neurons barely move from their initial positions; the network acts like a fixed linear model. In contrast, the mean-field regime allows for true feature learning.
As Taiji Suzuki and Denny Wu argue in Feature learning via mean-field Langevin dynamics, this allows networks to learn complex structures, like "sparse parities," that kernel methods find impossible. The Wasserstein flow allows the distribution of neurons to adapt its shape to the data, effectively "discovering" the most relevant features during training.
A Unified Language for Modern AI
The Wasserstein gradient flow framework is proving to be a universal language for diverse AI architectures:
- Reinforcement Learning: Andréa Agazzi uses this framework in Global optimality of softmax policy gradient to prove that policy gradients in wide networks converge to optimal policies.
- Generative Models: Gabriel Peyré highlights in Optimal and Diffusion Transports in Machine Learning how this geometry bridges the gap between GANs and Diffusion models.
- Transformers: Recent work by Juno Kim and Taiji Suzuki in Transformers Learn Nonlinear Features In Context applies mean-field dynamics to the attention landscape, explaining how Large Language Models learn nonlinear features during in-context learning.
By viewing training as a continuous flow of probability mass, researchers are finally moving past the "black box" of neural optimization toward a rigorous, fluid-dynamic theory of how machines learn.