The Wasserstein distance is a powerful tool for comparing probability distributions, but its computational cost in high dimensions is often prohibitive. The Sliced Wasserstein (SW) distance offers a clever bypass: instead of solving a complex optimal transport problem in $D$ dimensions, we project the data onto a set of one-dimensional lines (slices), where the problem has a simple closed-form solution.
However, standard SW distance relies on a "blind" average across all possible projection directions. In high-dimensional spaces, most random directions are uninformative, capturing little of the actual discrepancy between distributions. Recent research has shifted toward adaptive importance weighting, a strategy that transforms the SW distance from a passive average into a targeted, discriminative metric.
The Problem with Uniform Slicing
In a typical Sliced Wasserstein calculation, we sample directions $\theta$ uniformly from a unit sphere. As Hongliang Zhang and Shuo Chen point out in their work on Discriminative Spherical Sliced-Wasserstein, this uniform weighting is often suboptimal. If two distributions differ significantly along only a few axes, a uniform average dilutes that signal with noise from the hundreds of other directions where the distributions look identical.
This "blurriness" leads to two main issues:
- Low Discriminative Power: The metric may fail to distinguish between subtly different distributions.
- Slow Convergence: In tasks like gradient flow or generative modeling, the optimizer wastes time following "noisy" gradients from uninformative slices.
Weighting as a Spotlight
The core intuition behind the new wave of SW variants is to treat the projection process as an importance sampling problem. Instead of giving every direction $\theta$ an equal vote, we assign a weight $w(\theta)$ based on how much "discrepancy" that direction reveals.
Khai Nguyen and Nhat Ho formalized this in the Energy-Based Sliced Wasserstein (EBSW) distance. They define the projection distribution as an energy-based model where the probability of choosing a direction is proportional to the distance measured along that direction. By applying an exponential energy function, they create a "soft-max" effect: the metric automatically focuses its attention on the most informative features of the data without requiring an expensive optimization loop to find them.
Mechanisms for Finding the Best Slices
Researchers have proposed several ways to determine these weights $w(\theta)$ efficiently:
- Energy and Distance: In EBSW, the weight is a function of the 1D Wasserstein distance itself. This makes the metric self-correcting—the more a direction matters, the more it is weighted.
- Random-Path Projections: Khai Nguyen and colleagues also introduced Random-Path Projection SW (RPSW), which picks directions based on the paths between samples from the two distributions. This ensures the slices are inherently tied to the actual geometry of the data difference.
- Reservoir Sampling: Borrowing from computer graphics, Andreas Engelhardt and Mark Boss developed ReSWD. This method uses a "memory" (a reservoir) to keep track of directions that yielded high distances in previous iterations, drastically reducing gradient noise during training.
- Geometric Proximity: For complex structures like trees, Hoang V. Tran and Tam Le proposed Distance-Based Tree-Sliced Wasserstein, which weights structural components based on the geometric distance between data points and projection lines.
Why Weighting Accelerates Learning
A crucial theoretical insight comes from Da Xu and Yuting Ye in their study of importance weighting for deep learning. They found that in over-parameterized models, while importance weights might not change the final "asymptotic" state of a model (the max-margin solution), they profoundly impact the speed of convergence.
In the context of Sliced Wasserstein, this means that while a uniform SW and a weighted SW might eventually point toward the same global minimum, the weighted version reaches it much faster. By prioritizing directions with larger one-dimensional distances, the metric provides a cleaner, stronger signal to the optimizer. This is why Tung Le and Xiaohui Xie can use these techniques to achieve state-of-the-art unsupervised shape correspondence—the importance sampling allows the model to align complex geometric features that would be lost in the noise of uniform slicing.
By moving from uniform averaging to data-adaptive weighting, the Sliced Wasserstein distance evolves from a simple approximation into a precise, efficient tool for high-dimensional distribution alignment.