In reinforcement learning (RL), we are almost always in the business of comparing probability distributions. Whether we are trying to make an agent's current policy close to an old policy (trust-region methods), forcing an ensemble of agents to explore different areas of an environment (diversity), or matching an agent's behavior to an expert's demonstrations (imitation learning), we need a mathematical yardstick to measure the distance between distributions.
Traditionally, the field has relied on information-theoretic measures like the Kullback-Leibler (KL) divergence or the Total Variation (TV) distance. However, these metrics suffer from a fundamental flaw when applied to physical or behavioral spaces: they are entirely blind to the underlying geometry of the space.
A growing body of research addresses this limitation by replacing traditional divergences with the Wasserstein distance (also known as the Earth Mover's Distance). By leveraging the geometric structure of state and action spaces, Wasserstein-based metrics provide smooth, persistent gradients that fundamentally improve policy optimization, exploration, and imitation learning.
The Problem with Information-Theoretic Divergences
To understand why Wasserstein metrics are so effective, consider an intuitive example. Imagine an agent exploring a gridworld environment. Policy $A$ always visits the top-left corner, while Policy $B$ always visits the bottom-right corner.
If we compute the KL divergence or TV distance between the state occupancy distributions of Policy $A$ and Policy $B$, these metrics yield a harsh truth: the distributions have completely disjoint supports (they do not overlap). Consequently, the KL divergence explodes to infinity, and the TV distance saturates at its maximum value.
Crucially, if we slightly shift Policy $A$ one step closer to the bottom-right corner, the KL divergence remains infinite and the TV distance remains saturated. Because these metrics only look at point-by-point overlaps in probability mass, their gradients with respect to the policy parameters become zero or completely uninformative. The optimization algorithm knows the policies are different, but it receives no directional signal on how to change Policy $A$ to make it look more like Policy $B$.
Earth Mover's Distance: Geometry Matters
The Wasserstein distance resolves this issue by treating probability distributions not as abstract functions, but as physical piles of dirt. The distance is defined as the minimum "work" (mass multiplied by transport distance) required to reshape one probability distribution into another.
Because the definition explicitly includes a ground metric—the actual distance between points in the state or action space—it remains highly sensitive to shifts even when distributions are completely disjoint. If Policy $A$ is 10 steps away from Policy $B$, the Wasserstein distance reflects that physical separation. If Policy $A$ moves one step closer, the Wasserstein distance decreases, providing a smooth, continuous gradient that an optimization algorithm can follow.
Smooth Policy Optimization and Gradient Flows
This geometric awareness translates directly into more stable and robust policy updates. Instead of treating policy optimization as a search through ungrounded parameter spaces, Ruiyi Zhang et al. re-imagine the learning process as a Wasserstein Gradient Flow (WGF). By updating policies within a trust region defined by the Wasserstein distance, the optimization trajectory moves smoothly along the manifold of possible distributions, avoiding the erratic jumps and premature convergence common to standard policy gradients.
Furthermore, traditional policy gradients evaluate updates based purely on raw action probabilities, ignoring what those actions physically achieve in the environment. To fix this, Ted Moskovitz et al. introduce Wasserstein Natural Gradients (WNG). By mapping agent trajectories directly onto a behavioral manifold and measuring distances via optimal transport, WNG ensures that gradient updates reflect the actual physical consequences of policy changes, dramatically improving stability in complex environments.
This geometric smoothing also yields powerful theoretical guarantees. In environments where transitions are smooth (Lipschitz continuous), Marcello Restelli et al. demonstrate that using the $L_1$-Wasserstein metric to bound state distribution shifts allows for the calculation of optimal step sizes. Unlike TV-based bounds, which collapse in deterministic or non-overlapping settings, the Wasserstein metric guarantees monotonic performance improvements across a much wider range of environments.
Driving Exploration and Behavioral Diversity
If the Wasserstein distance provides a smooth gradient when distributions do not overlap, it is uniquely suited for the exact opposite task: pushing distributions apart to force exploration.
In unsupervised skill discovery, agents try to learn a diverse set of behaviors without external rewards. Traditional methods maximize the mutual information between states and skills using KL-like objectives, which often causes skills to clump together in easily distinguishable local regions. Yucheng Yang et al. show that replacing KL divergence with the Wasserstein distance creates a true metric space that satisfies the triangle inequality. Optimizing this metric naturally prevents skill clustering and aggressively pushes skill distributions toward the outer boundaries of the reachable state space.
At the single-agent level, Ishan Durugkar leverages this property for intrinsic control (WIC) by maximizing the Wasserstein-1 distance between an agent's starting state and its subsequently visited states. This simple geometric incentive forces the agent to break out of local bottlenecks and explore deep into the environment's state space.
Traditional Divergence (KL / TV):
[Policy A] <-- Infinite Distance / Zero Gradient --> [Policy B]
(No signal on how to move closer or further apart)
Wasserstein Distance (Metric-Aware):
[Policy A] <------- Smooth, Continuous Gradient -------> [Policy B]
(Work-based transport cost provides a persistent directional signal)
This persistent gradient is equally valuable when training populations or ensembles of agents:
- Continuous Action Ensembles: When generating diverse game levels, Ziqi Wang et al. use the 2-Wasserstein distance as a differentiable regularization tool to directly penalize similarity between the continuous action distributions of ensemble members, forcing them to discover qualitatively distinct strategies.
- Multi-Agent Trajectories: In cooperative multi-agent tasks, the authors of Encouraging metric-aware diversity in contrastive representation space maximize the Wasserstein distance between agent trajectory distributions in a contrastive latent space. This prevents agents from becoming behavioral clones of one another, maintaining a smooth exploratory pressure throughout training.
- Distributional RL Exploration: Even within value-based methods, Moritz Akiya Zanger et al. demonstrate that calculating the 1-Wasserstein distance between the predicted return distributions of an ensemble provides a powerful exploration bonus that captures epistemic uncertainty far better than simple scalar variance.
Reward-Free Evaluation and Imitation Learning
Beyond optimization and exploration, the Wasserstein distance serves as an exceptionally reliable diagnostic tool because it accurately quantifies how closely an agent's global behavior matches a target distribution.
In imitation learning from observations (LfO), an agent attempts to match an expert's state occupancy without access to expert actions. Alex Schwing et al. optimize the primal form of the Wasserstein distance to perform state occupancy matching, allowing the integration of flexible, contrastively learned distance metrics that capture the underlying environment geometry better than rigid Euclidean distances. To guide policy optimization at a trajectory level, Aldo Pacchiano et al. utilize the dual formulation of the Wasserstein distance to derive learned score functions. These scores evaluate individual trajectories based on their global contribution to the behavioral distribution, acting as a smooth secondary reward signal.
Perhaps the most practical validation of Wasserstein metrics as a true reflection of behavior comes from hyperparameter tuning. In real-world imitation learning, ground-truth reward functions are completely absent, making it notoriously difficult to select hyperparameters or know when to stop training.
Leonard Hussenot et al. systematically demonstrated that the Wasserstein distance between an agent's state distribution and the expert's demonstration distribution acts as a robust, reward-free proxy for the true underlying reward. Because it tracks the global alignment of trajectories rather than brittle, step-by-step action matching, minimizing the Wasserstein distance allows practitioners to reliably select optimal hyperparameters and perform early stopping without ever peeking at an external reward signal.
Key Foundational Work
Policy Optimization as Wasserstein Gradient Flows (2018) — contributed 1 concept elements to this direction.
Policy gradient in Lipschitz Markov Decision Processes (2015) — contributed 1 concept elements to this direction.