In standard machine learning optimization, we treat parameters as points in a flat, Euclidean space. Moving from an old parameter to a new one is as simple as drawing a straight line. However, when our optimization objective requires us to manipulate entire probability distributions—such as in variational inference, generative modeling, or calculating structural averages—this flat assumption fails.
If we average two sharp Gaussian distributions using standard Euclidean metrics ($L^2$), we get a flattened mixture with two half-height peaks. To preserve the underlying geometry of the data, we must instead use the Wasserstein metric, which measures the minimal cost of physically transporting mass from one distribution to another. Under this metric, the "average" (or barycenter) is a single sharp peak positioned right in the middle.
While intuitively appealing, optimizing directly within the Wasserstein space is notoriously difficult. The space of probability measures is an infinite-dimensional, curved Riemannian-like manifold. This curvature introduces severe optimization hurdles: standard linear operations do not exist, functionals that look simple can become highly non-convex, and optimization landscapes become riddled with local minima and saddle points.
A rapidly growing body of research is overcoming these hurdles by uncovering the hidden geometric properties of Wasserstein space and designing algorithms that natively respect its curvature.
The Challenge of Curvature and Non-Convexity
To understand why Wasserstein optimization is difficult, consider how we define a "straight line." In a flat space, a line is a linear interpolation. In Wasserstein space, the shortest path between two distributions is a geodesic—a continuous morphing that slides mass along optimal transport routes.
Convexity defined along these geodesics is known as displacement convexity. As Rentian Yao et al. point out, displacement convexity is exceptionally rare and difficult to verify in common statistical models. Without it, standard convergence proofs break down. Simple linear functionals can appear deceptively non-convex when analyzed through mass-moving geometries (Pavel Dvurechensky et al.).
Furthermore, the geometry itself can introduce ambiguities:
- Geodesic Non-Uniqueness: In spaces governed by the 1-Wasserstein distance, multiple shortest paths can exist between the same two distributions, preventing optimization algorithms from reliably converging to a unique optimal mapping (Jacob Leygonie et al.).
- Non-Unique Critical Points: When computing population barycenters (Karcher means), the inherent non-convexity means that unique critical points are not guaranteed in general, complicating the development of stable online algorithms (Joaquin Fontbona et al.).
This "bumpy" energy landscape means that standard first-order optimizers frequently get trapped or fail to minimize Wasserstein-based regularizers effectively (Okan Koç et al.; Wasserstein Dictionaries of Persistence Diagrams).
Hidden Structure: The Polyak-Łojasiewicz Inequality
If the landscape is non-convex, why do simple first-order methods like Gradient Descent (GD) and Stochastic Gradient Descent (SGD) often work well in practice? The answer lies in global geometric conditions that act as proxies for convexity.
The most powerful of these is the Polyak-Łojasiewicz (PL) inequality. In Euclidean space, a function satisfies the PL inequality if the square of its gradient bounds its suboptimality from above. Crucially, this condition does not require the function to be convex, yet it guarantees that every stationary point is a global minimum, allowing gradient descent to achieve fast, exponential convergence.
Recent work has successfully translated this insight into curved spaces. Sinho Chewi et al. proved that despite the non-geodesic convexity of the Wasserstein barycenter objective, GD and SGD achieve fast, global convergence for Gaussian distributions because the objective satisfies a Wasserstein variant of the PL inequality.
[Non-Convex Landscape] ──(PL Inequality Satisfied)──> [Every Stationary Point is Global] ──> [Fast GD/SGD Convergence]
Similarly, minimizing the Wasserstein distance between a continuous distribution and a discrete point cloud is highly non-convex. Yet, Quentin Mérigot et al. proved a modified PL inequality for this setting, providing the first non-asymptotic convergence guarantees for practical quantization algorithms like Lloyd's method.
Flattening the Space via Coordinate Transformations
Another strategy for bypassing Wasserstein curvature is to find a coordinate transformation that maps the curved manifold into a flat Hilbert space where standard calculus applies.
On the real line ($\mathbb{R}$), this mapping is exact. By transforming probability measures into their corresponding quantile functions, the optimal transport problem simplifies completely: the Wasserstein distance becomes a standard $L^2$ distance between quantiles, and Wasserstein geodesics become straight lines. Johannes Hertrich et al. leveraged this property to derive explicit analytic formulas for Wasserstein gradient flows under Maximum Mean Discrepancy (MMD) targets, turning an otherwise intractable manifold problem into a solvable differential equation.
Curved Space: [Probability Measure μ] ──(Wasserstein Geodesic)──> [Probability Measure ν]
│ │
(Quantile Transformation) (Quantile Transformation)
▼ ▼
Flat Hilbert Space: [Quantile Function q_μ] ───────(Straight Line)───────> [Quantile Function q_ν]
In higher dimensions, exact flattening is impossible. However, we can exploit structured sub-spaces:
- The Gaussian Subspace: For Gaussian distributions, the Wasserstein metric simplifies to the Bures-Wasserstein distance. Nina Vesseron et al. utilized the "lifted" geometry of invertible matrices to perform exact Geodesic Principal Component Analysis (GPCA), capturing intrinsic modes of distribution variation without linear distortion.
- Hilbert Space Proxies: Rather than optimizing directly on the manifold of measures, Gudmund Pammer et al. shifted Martingale Optimal Transport problems into Hilbert spaces, creating a convergent $L^2$ gradient flow for constructing elusive Bass martingales.
Translating Advanced Optimizers to Tangent Spaces
To run gradient descent directly on a manifold, you cannot simply add a vector to a point. Instead, you must compute a gradient in the tangent space of the current distribution and push the distribution along a geodesic path—a process known as a Wasserstein gradient flow.
To accelerate these flows, researchers are generalizing advanced Euclidean optimization frameworks to discrete-time Wasserstein algorithms:
Mirror Descent and Preconditioning
In flat spaces, Mirror Descent adapts to the local geometry of an objective using a Bregman divergence. Clément Bonet et al. successfully generalized both Mirror Descent and Preconditioned Gradient Descent to the infinite-dimensional Wasserstein space. By tailoring the optimization geometry to the problem, they established rigorous discrete-time convergence rates, enabling massive speedups in real-world applications like single-cell biological data alignment.
Escaping Saddle Points
In Euclidean optimization, we escape saddle points by adding isotropic noise. In Wasserstein space, however, arbitrary noise can destroy the structure of the distribution. Naoya Yamamoto et al. solved this by introducing the Perturbed Wasserstein Gradient Flow (PWGF). This method injects noise guided explicitly by the "Wasserstein Hessian" via a Gaussian process, providing a mathematically rigorous way to escape saddle points on infinite-dimensional, non-convex measure landscapes.
Majorization-Minimization
When direct optimization is too costly, Majorization-Minimization (MM) steps in by iteratively optimizing a simpler, locally tight surrogate function. Chengliang Tang et al. combined flexible Gaussian mixture models with a novel MM framework, making distribution-to-distribution prediction computationally practical and theoretically sound.
Broadening the Framework
The mathematical machinery developed for Wasserstein geometry is fundamentally rewriting convergence frameworks across statistics and machine learning.
For instance, proving how fast a Bayesian posterior distribution narrows down (contracts) typically relies on neighborhood conditions that are completely incompatible with the Wasserstein metric. Anirban Bhattacharya et al. bypassed this limitation by leveraging a simple histogram model, proving minimax optimal posterior contraction rates under Wasserstein distance across all dimensions.
Similarly, the concept of dynamical stability is expanding. Standard definitions of monotonicity—used to prove that a system will settle into a steady state—require linear vector spaces. Lauren Conger et al. successfully extended monotonicity to the Wasserstein-2 metric space, providing a framework to guarantee that interacting, multi-species population flows will converge to a unique, stable Nash equilibrium.
Whether optimizing mean-field variational inference over structured polyhedral subsets (Yiheng Jiang et al.) or tracking the exact trajectories of continuous measures, understanding the geometric constraints of Wasserstein space is turning historically intractable problems into provably convergent, practical algorithms.
Key Foundational Work
Gradient descent algorithms for Bures-Wasserstein barycenters (2020) — contributed 3 concept elements to this direction.
Second-Order Models for Optimal Transport and Cubic Splines on the Wasserstein Space (2019) — contributed 1 concept elements to this direction.
Adversarial Computation of Optimal Transport Maps (2019) — contributed 1 concept elements to this direction.