Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution

Khai Nguyen, Nhat Ho

Abstract

The conventional Sliced Wasserstein (SW) distance is defined between probability measures over vector spaces. When applied to images, standard practice involves vectorizing the images and projecting them onto one-dimensional spaces via matrix multiplication. However, this approach suffers from two primary limitations: the destruction of spatial structures and extreme memory inefficiency due to the size of projection vectors. To address these, we propose Convolution Sliced Wasserstein (CSW), which replaces traditional vectorization with convolution operators. By incorporating stride, dilation, and non-linear activations, CSW preserves local image structures and significantly reduces the parameter count. Our investigation covers the metricity, sample complexity, and computational efficiency of CSW, demonstrating its superior performance in comparing image distributions and training deep generative models.

Introduction

Measuring the discrepancy between high-dimensional probability distributions is a fundamental task in machine learning, particularly in generative modeling. The Wasserstein distance, rooted in Optimal Transport theory, provides a robust metric for this purpose but is computationally prohibitive for large datasets. The Sliced Wasserstein (SW) distance offers a scalable alternative by projecting high-dimensional measures into one dimension, where the Wasserstein distance has a closed-form solution.

Despite its popularity, SW is poorly suited for image data. Because it requires images to be flattened into vectors, the spatial relationships between pixels—the very features that define an image—are ignored. Furthermore, each slicing direction requires a vector of the same dimension as the image, leading to massive memory overhead as resolution increases. This paper introduces a convolutional framework for slicing that treats images as structured tensors rather than flat vectors.

The Convolutional Slicer

The core innovation of this work is the replacement of the linear projection step with a sequence of convolution operations. In standard SW, an image $X$ is projected using a random vector $\theta$. In CSW, the image undergoes a series of convolutions that progressively reduce its spatial dimensions until a scalar value is reached.

Variants of CSW

To optimize performance and capture different types of visual features, several variants of the convolution slicer were developed:

  • CSW-base (CSW-b): Uses standard convolutions to reduce image size incrementally.
  • CSW-stride (CSW-s): Employs strided convolutions to downsample the image more aggressively, increasing the receptive field while maintaining a low parameter count.
  • CSW-dilation (CSW-d): Utilizes dilated convolutions to capture long-range spatial dependencies without increasing the number of parameters.
  • Non-linear CSW (NCSW): Introduces activation functions (e.g., Sigmoid) between layers, allowing the slicer to capture non-linear patterns that linear projections might miss.
  • Max-CSW: Instead of averaging over random slices, this variant uses optimization to find the specific convolutional slice that maximizes the distance between distributions, providing a more sensitive measure of discrepancy.

Theoretical and Practical Advantages

Parameter Sharing and Memory Efficiency

By using small convolution kernels (e.g., $3 \times 3$) instead of full-image-sized projection matrices, CSW achieves dramatic memory savings. For instance, on the MNIST dataset ($28 \times 28$ pixels), a standard SW slice requires 784 parameters. In contrast, a CSW-s slicer requires only 57 parameters. This efficiency becomes even more pronounced as image resolution scales to $128 \times 128$ or higher.

Spatial Awareness

Convolutions are inherently designed to process local neighborhoods. This allows CSW to detect edges, textures, and shapes more effectively than the "spatially blind" vectorization used in standard SW. This structural awareness translates to more meaningful distance measurements in the context of computer vision.

Sample Complexity

A critical theoretical concern for any distance metric is how much data is required to estimate it accurately. We prove that CSW maintains a sample complexity of $O(n^{-1/2})$, meaning it avoids the "curse of dimensionality" and remains effective even as the resolution of the images increases.

Experimental Results

We evaluated CSW against standard SW across several benchmarks, including MNIST, CIFAR10, STL10, and CelebA-HQ.

Generative Modeling

When used as a loss function for training Generative Adversarial Networks (GANs), CSW variants consistently outperformed standard SW. On the CIFAR10 dataset, using 1,000 projections, CSW-s achieved a Fréchet Inception Distance (FID) of 36.80, a significant improvement over the 44.25 achieved by vanilla SW. Qualitatively, the images generated via CSW were sharper and exhibited fewer structural artifacts.

Computational Performance

While the convolutional operations introduce a slight overhead in raw computation time per slice, the improved convergence rates and memory efficiency often result in faster overall training cycles. CSW-s and CSW-d, in particular, provide a favorable trade-off between speed and generative quality.

Conclusion

Convolution Sliced Wasserstein represents a significant step forward in adapting Optimal Transport tools for structured data. By leveraging the inductive biases of convolutions, CSW addresses the spatial blindness and memory bottlenecks of traditional Sliced Wasserstein distances. While currently a pseudo-metric, empirical evidence suggests it behaves as a true metric in practice. Future work will focus on formalizing the metricity of CSW and extending its application to other structured domains such as 3D point clouds and video sequences.

Summary updated

Abstract

The conventional sliced Wasserstein is defined between two probability measures that have realizations as vectors. When comparing two probability measures over images, practitioners first need to vectorize images and then project them to one-dimensional space by using matrix multiplication between the sample matrix and the projection matrix. After that, the sliced Wasserstein is evaluated by averaging the two corresponding one-dimensional projected probability measures. However, this approach has two limitations. The first limitation is that the spatial structure of images is not captured efficiently by the vectorization step; therefore, the later slicing process becomes harder to gather the discrepancy information. The second limitation is memory inefficiency since each slicing direction is a vector that has the same dimension as the images. To address these limitations, we propose novel slicing methods for sliced Wasserstein between probability measures over images that are based on the convolution operators. We derive convolution sliced Wasserstein (CSW) and its variants via incorporating stride, dilation, and non-linear activation function into the convolution operators. We investigate the metricity of CSW as well as its sample complexity, its computational complexity, and its connection to conventional sliced Wasserstein distances. Finally, we demonstrate the favorable performance of CSW over the conventional sliced Wasserstein in comparing probability measures over images and in training deep generative modeling on images.
BibTeX
@inproceedings{nguyen2022revisiting,
  author = {Khai Nguyen and Nhat Ho},
  title = {{Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=O4Q39aQFz0Y},
}
Figure