Introduction
The Distributional Sliced-Wasserstein (DSW) distance is a novel metric for comparing probability distributions that bridges the gap between the computationally expensive Optimal Transport and the overly simplistic Max-Sliced Wasserstein. By learning an optimal distribution of projection directions rather than relying on a single direction or a random set, DSW achieves state-of-the-art performance in generative modeling with significantly fewer projections and faster convergence.
The Problem with Existing Distances
Measuring the distance between high-dimensional probability distributions is a cornerstone of modern machine learning, particularly for training Generative Adversarial Networks (GANs). The "gold standard" is the Wasserstein distance (derived from Optimal Transport), but it is computationally brutal, scaling poorly as the number of data points increases.
To solve this, researchers often use Sliced-Wasserstein (SW) distances. The idea is simple: project high-dimensional data onto a 1D line (a "slice"), where the Wasserstein distance is trivial to compute, and then average the results over many random directions. However, SW has two major flaws:
- Standard SW requires thousands of random projections to be accurate, many of which are "irrelevant" directions that don't help distinguish the distributions.
- Max-SW tries to find the single best projection, but this ignores the complex, multi-dimensional structure of the data.
This paper asks: Can we find a small, optimal set of directions that captures the most important differences between two distributions?
The DSW Approach
The authors introduce Distributional Sliced-Wasserstein (DSW). Instead of picking directions at random or picking just one, DSW searches for an optimal probability distribution ($\sigma$) over the unit sphere of projections.
Key technical components include:
- Regularization: To prevent all learned directions from collapsing into a single point (which would just be Max-SW), the authors add a constraint. This constraint forces the distribution to favor directions that are far apart (ideally orthogonal), ensuring the model "explores" different dimensions of the data.
- Neural Reparameterization: Since searching for a distribution over a sphere is difficult, they use a deep neural network to parameterize a "push-forward" map. This network takes a uniform distribution as input and transforms it into the optimal distribution of directions.
- Dual Formulation: They derive a dual form of the distance that allows them to optimize the projection-finding network using standard stochastic gradient ascent.
Key Insights
- The "Middle Way": DSW treats SW and Max-SW as special cases. SW is a uniform distribution over directions; Max-SW is a single "delta" point. DSW finds the sweet spot in between.
- Orthogonality is Key: By forcing projections to be distinct through the regularization parameter ($\lambda_C$), the model avoids redundant information. As regularization increases, the learned directions become nearly orthogonal.
- Beating the Curse of Dimensionality: The authors prove that DSW is a proper metric and maintains a statistical error rate that doesn't explode as dimensions increase, making it theoretically sound for high-dimensional image data.
- Efficiency: DSW-10 (using only 10 learned projections) often outperforms standard SW using 10,000 random projections.
Key Results
The researchers evaluated DSW on several standard generative modeling benchmarks, including MNIST, CIFAR10, CelebA, LSUN, and Color Transfer tasks.
- Speed and Convergence: DSW helps models converge much faster. On MNIST, DSW-10 achieved better model quality than Max-SW while being an order of magnitude faster.
- Generative Quality: On the LSUN dataset, DSW achieved a Fréchet Inception Distance (FID) of 46 with only 100 projections. In comparison, standard SW with 10,000 projections still had a worse FID of over 60.
- Visual Fidelity: Qualitatively, images generated using DSW are sharper and more recognizable than those generated with previous sliced-based methods, which often produce blurrier results due to "noisy" random projections.
Limitations and Future Work
- Hyperparameter Tuning: The performance of DSW depends on the regularization parameter $\lambda_C$. Currently, this needs to be tuned empirically for different datasets. Future work should investigate a more theoretical way to choose this value.
- Computational Overhead: While DSW requires fewer projections, there is an added cost to training the neural network that finds those projections. While this is a net win for large-scale tasks, it adds a layer of complexity to the implementation.
- Non-linear Extensions: While the paper touches on "Generalized" DSW (using non-linear projections), most of the heavy lifting is done with linear slices. Exploring more complex, non-linear slicing functions remains a promising frontier.
Summary updated