Neural Network Training as Wasserstein Gradient Flow

This direction is split into other directions: Wasserstein gradient flow. Its last version stays readable and citable here.

Version 1 of 159 papers66 conceptsfrozen 2026-10-04

In traditional machine learning, we view a neural network as a collection of discrete weights—individual "particles" that we nudge around a high-dimensional landscape to minimize a loss function. However, as networks grow wider, this particle-based view becomes mathematically cumbersome. A more powerful perspective, championed by researchers like Weinan E, Chao Ma, and Lei Wu in Machine Learning from a Continuous Viewpoint, treats the network as a continuous distribution of neurons.

By shifting from "weights as points" to "weights as a density," the chaotic, non-convex optimization of deep learning reveals a hidden, elegant structure: a Wasserstein gradient flow.

From Particles to Fluid Dynamics

When a neural network is infinitely wide, we no longer care about the specific value of the 42nd neuron. Instead, we care about the distribution of all neurons. This is the "mean-field" regime. As Song Mei, Andrea Montanari, and colleagues demonstrate in A Mean Field View of the Landscape of Two-Layers Neural Networks, the training of these networks can be modeled as the evolution of a probability measure.

Imagine the neurons as a cloud of gas. Gradient descent isn't just moving points; it is a force field that deforms and shifts this cloud. The math that describes this "mass-moving" process is Optimal Transport. Specifically, the optimization follows a gradient flow in the Wasserstein space—a geometric space where the distance between two distributions is the minimum "work" required to transform one into the other.

Why Global Convergence is Possible

The most striking result of this viewpoint is that it explains why over-parameterized networks find global optima. While the loss landscape is notoriously non-convex with respect to individual weights, it often becomes convex (or "displacement convex") with respect to the distribution of those weights.

Lénaïc Chizat and Francis Bach show in On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport that in the many-particle limit, the gradient flow in Wasserstein space reliably converges to a global minimizer. By "lifting" the problem into the infinite-dimensional space of measures, the traps and local minima of the finite-dimensional landscape effectively disappear.

Feature Learning vs Lazy Training

This mean-field perspective is distinct from the "Neural Tangent Kernel" (NTK) or "lazy training" regime. In the NTK regime, neurons barely move from their initial positions; the network acts like a fixed linear model. In contrast, the mean-field regime allows for true feature learning.

As Taiji Suzuki and Denny Wu argue in Feature learning via mean-field Langevin dynamics, this allows networks to learn complex structures, like "sparse parities," that kernel methods find impossible. The Wasserstein flow allows the distribution of neurons to adapt its shape to the data, effectively "discovering" the most relevant features during training.

A Unified Language for Modern AI

The Wasserstein gradient flow framework is proving to be a universal language for diverse AI architectures:

By viewing training as a continuous flow of probability mass, researchers are finally moving past the "black box" of neural optimization toward a rigorous, fluid-dynamic theory of how machines learn.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_network_training_as_wasserstein_g_4a1bac2e,
  title = {Neural Network Training as Wasserstein Gradient Flow},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-network-training-as-wasserstein-gradient-flow/rdr_4a1bac2e3f5e4a51988741b105fe06c6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_network_training_as_wasserstein_g_4a1bac2e_v1,
  title = {Neural Network Training as Wasserstein Gradient Flow},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-network-training-as-wasserstein-gradient-flow/rdr_4a1bac2e3f5e4a51988741b105fe06c6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}