Abstract
Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in recent years due to their fast computation and scalability even when probability measures lie in a very high-dimensional space. However, SW requires many unnecessary projection samples to approximate its value, while Max-SW only uses the most important projection, ignoring information from other useful directions. To address these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal distribution over projections to balance between exploring distinctive projecting directions and the informativeness of the projections themselves. We show that DSW is a generalization of Max-SW and can be computed efficiently by searching for the optimal push-forward measure over a set of probability measures on the unit sphere, satisfying certain regularizing constraints that favor distinct directions. Finally, we conduct extensive experiments with large-scale datasets to demonstrate the favorable performance of the proposed distances over previous sliced-based distances in generative modeling applications.
Introduction
Measuring the distance between high-dimensional probability distributions is a fundamental task in machine learning, particularly for training Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). While Optimal Transport (OT) provides a powerful framework via the Wasserstein distance, its computational complexity ($O(k^3)$ or $O(k^2)$ with approximations) makes it prohibitive for large-scale applications.
To mitigate this, researchers often employ the Sliced-Wasserstein (SW) distance. SW projects high-dimensional data onto 1D lines (slices), where OT has a simple, fast closed-form solution. However, standard SW relies on uniform sampling, which often picks "irrelevant" directions in high dimensions. Conversely, Max-SW searches for a single optimal direction, which may ignore complex structural differences present in other dimensions. DSW bridges this gap by learning an optimal, diverse distribution of projection directions.
Methodology
The core innovation of DSW is treating the slicing process as a search for an optimal probability measure $\sigma$ over the unit sphere of possible directions. Unlike Max-SW, which collapses to a single point, DSW seeks a distribution that maximizes the projected Wasserstein distance while maintaining diversity among directions.
Diversity Regularization
To prevent learned directions from collapsing into a single point, the authors introduce a regularization term: $\mathbb{E}_{\theta, \theta' \sim \sigma} [|\theta^\top \theta'|] \le C$. This constraint forces the chosen directions to be as orthogonal as possible, ensuring that the model captures multiple informative facets of the data distribution.
Neural Parameterization
The search for the optimal distribution is parameterized using a deep neural network. This network learns to map uniform noise into the most "important" directions for a specific pair of distributions. The authors also extend this to Distributional Generalized Sliced-Wasserstein (DGSW), which utilizes non-linear projections to capture more complex data manifolds.
Experimental Evaluation
The authors evaluated DSW and DGSW across several standard generative modeling benchmarks, including MNIST, CIFAR10, CelebA, and LSUN (bedroom category).
Key Results
- Superior Generative Quality: On the LSUN dataset, DSW with only 100 projections achieved an FID score of 46, significantly outperforming standard SW, which scored over 60 even with 10,000 projections.
- Efficiency and Convergence: In MNIST experiments, DSW-10 (using only 10 projections) converged to a lower Wasserstein-2 error faster than all other slice-based baselines.
- Computational Trade-offs: While DSW requires an inner loop to train the projection-finding network, the massive reduction in the number of required projections makes it more efficient overall. DSW-100 is faster than SW-10000 while providing superior image quality.
Conclusion
DSW provides a probabilistic unification of existing sliced-based distances, showing that SW and Max-SW are extreme special cases (uniform vs. Dirac delta distributions). By focusing on a diverse set of informative projections, DSW overcomes the curse of dimensionality and the inefficiencies of random sampling. While the regularization weight $\lambda_C$ currently requires manual tuning, DSW represents a significant step forward in making Optimal Transport-based generative modeling both scalable and high-performing.
Summary updated