In traditional reinforcement learning and imitation learning, we often measure the "distance" between two behaviors using information-theoretic metrics like Kullback-Leibler (KL) divergence or Jensen-Shannon (JS) divergence. While these metrics are mathematically convenient, they suffer from a fundamental flaw in continuous control: they don't care about the underlying geometry of the action space.
If an expert performs an action at $x=1.0$ and an agent performs an action at $x=1.1$, a KL-based objective might treat this as a total failure (zero overlap), providing no useful gradient for improvement. A new wave of research suggests that by switching to the Wasserstein distance (also known as Earth Mover's distance), we can build agents that learn more stably, generalize better, and come with tighter theoretical guarantees.
The Geometry of Behavior
The core intuition behind using the Wasserstein distance is that it treats probability distributions as "piles of dirt." To transform one distribution into another, you have to move mass across the space. The cost depends on how much mass you move and how far you move it.
This "how far" is the game-changer. In Truly Deterministic Policy Optimization, Ehsan Saleh, Timothy Bretl, and colleagues point out that traditional metrics like KL divergence are often undefined for deterministic policies because their supports do not overlap. By using Wasserstein distance as a regularization metric, their TDPO framework can optimize deterministic policies directly. It leverages the actual distance in the action space to provide a meaningful gradient even when the agent's current guess is far from the target.
Stabilizing Adversarial Imitation
Adversarial imitation learning (like GAIL) typically functions like a GAN: a discriminator tries to distinguish between expert and agent trajectories, while the agent tries to fool the discriminator. However, standard GANs are notoriously unstable, often suffering from vanishing gradients when the discriminator becomes too good.
Researchers have found that the Wasserstein metric provides a much smoother optimization landscape:
- InfoGAIL: Yunzhu Li and Jiaming Song introduced InfoGAIL, which uses a Wasserstein GAN objective to handle high-dimensional visual inputs. This prevents the "mode collapse" common in standard imitation learning, allowing the agent to capture diverse expert styles.
- WDAIL: Similarly, Ming Zhang and Xiaoteng Ma developed WDAIL, demonstrating that replacing JS divergence with Wasserstein distance provides continuous, differentiable gradients. This prevents the training process from stalling and leads to much higher sample efficiency in robotic tasks.
- Behavioral Inference: Lina Yao and Xianzhi Wang applied these principles to Inverse Reinforcement Learning, using the Earth Mover's distance to infer user tendencies in complex environments like traffic control and recommendation systems without needing manually defined reward functions.
Why It Works: Hölder Continuity
Beyond empirical stability, the shift to Wasserstein metrics allows for more rigorous theory. In Tight Performance Guarantees of Imitator Policies with Continuous Actions, Alberto Maria Metelli and his team proved that the performance gap between an expert and an imitator is bounded by the Wasserstein distance between their policies.
The "secret sauce" here is Hölder continuity. Because the value functions in most physical environments change smoothly as the state or action changes, the Wasserstein distance—which respects that smoothness—provides a much tighter bound on the expected return than "all-or-nothing" metrics like Total Variation. This bridges the gap between the theoretical bounds of Behavioral Cloning and its actual success in real-world robotics.
Summary
The transition from information-theoretic divergences to geometric distances represents a shift in how we think about "errors" in AI behavior. By acknowledging that an action at $x=1.1$ is "closer" to $x=1.0$ than $x=5.0$ is, Wasserstein-based methods provide the smooth, consistent feedback necessary for training robust agents in the continuous, messy reality of the physical world.