Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution

Khai Nguyen, Nhat Ho

TL;DR

This paper introduces Convolution Sliced Wasserstein (CSW), a novel approach to measuring distances between image distributions. By replacing traditional vectorization and matrix multiplication with convolution operators, CSW preserves critical spatial structures and significantly reduces memory overhead, leading to faster convergence and higher-quality results in deep generative modeling.

The Problem with Standard Sliced Wasserstein on Images

The Wasserstein distance is a powerful tool for comparing probability distributions, but it is notoriously expensive to compute and suffers from the "curse of dimensionality." Sliced Wasserstein (SW) distances offer a workaround by projecting high-dimensional distributions onto one-dimensional lines, where the distance is easy to calculate.

Conventional Sliced Wasserstein process requiring image vectorization

However, when applied to images, standard SW has a major flaw: it requires vectorization. To "slice" an image, you must first flatten it into a long vector, destroying the 2D spatial relationships (like edges and textures) that define the image. Furthermore, in high-resolution settings, storing these projection vectors becomes a memory nightmare—a single projection for a $128 \times 128$ image requires a vector of the exact same size. This paper aims to bridge the gap between the efficiency of slicing and the inherent spatial nature of image data.

Methods: The Convolution Slicer

To address the limitations of vectorization, the authors propose replacing the standard Radon transform (matrix multiplication) with a Convolution Slicer.

Convolution Sliced Wasserstein process using convolution kernels

Instead of one giant projection vector, CSW uses a sequence of small, learnable (or randomly sampled) convolution kernels to progressively map an image down to a scalar value. They introduce three main variants of the CSW:

  1. CSW-base (CSW-b): Uses standard convolutions to reduce image size.
  2. CSW-stride (CSW-s): Incorporates strided convolutions to downsample more efficiently.
  3. CSW-dilation (CSW-d): Uses dilated convolutions to increase the receptive field, allowing the slicer to capture global information without increasing the number of parameters.

The authors also explore Non-linear CSW, which adds activation functions between convolution layers to capture more complex, non-linear features of the data. Mathematically, they prove that CSW is a pseudo-metric and maintains the same favorable sample complexity as standard SW, meaning it avoids the curse of dimensionality.

Key Insights and Results

The researchers evaluated their methods on standard computer vision benchmarks, including MNIST, CIFAR10 ($32 \times 32$), CelebA ($64 \times 64$), CelebA-HQ ($128 \times 128$), and STL10 ($96 \times 96$).

  • Spatial Awareness is Key: By using convolutions, the "slices" are forced to respect local pixel correlations. This makes the resulting one-dimensional projections more "discriminative," meaning they are better at catching the subtle differences between a real image and a blurry generated one.
  • The Memory Win: Standard SW memory scales with the total number of pixels ($O(cd^2)$). In contrast, the projection memory for CSW-stride and CSW-dilation scales much more favorably ($O(c + \log d)$). This makes CSW far more scalable for high-resolution image generation.
  • Better Image Quality & Faster Convergence: When integrated into a generative modeling framework, CSW outperformed vanilla SW. On CelebA-HQ ($128 \times 128$), CSW-s achieved a significantly lower FID score (a measure of image realism) of 22.25 compared to SW's 28.19 when using 1,000 projections. CSW variants also generally helped generative models converge in fewer epochs.
  • Performance with Fewer Slices: One of the most striking results was that CSW performs remarkably well even with very few projections ($L=1$). On CelebA, CSW-s with $L=1$ outperformed vanilla SW with $L=100$.

Limitations and Future Work

While CSW is highly effective for images, its current formulation is specifically tailored for grid-like data. Extending this "convolutional slicing" logic to other structured data, like graphs or point clouds, remains an open challenge.

Additionally, while the memory complexity is lower, the computational complexity of the "base" version (CSW-b) is actually higher than standard SW. While the stride and dilation versions mitigate this, there is still room to optimize the implementation for even faster training. The authors also suggest that further theoretical work could investigate the conditions under which CSW becomes a full metric rather than a pseudo-metric.

Summary updated

Abstract

The conventional sliced Wasserstein is defined between two probability measures that have realizations as \textit{vectors}. When comparing two probability measures over images, practitioners first need to vectorize images and then project them to one-dimensional space by using matrix multiplication between the sample matrix and the projection matrix. After that, the sliced Wasserstein is evaluated by averaging the two corresponding one-dimensional projected probability measures. However, this approach has two limitations. The first limitation is that the spatial structure of images is not captured efficiently by the vectorization step; therefore, the later slicing process becomes harder to gather the discrepancy information. The second limitation is memory inefficiency since each slicing direction is a vector that has the same dimension as the images. To address these limitations, we propose novel slicing methods for sliced Wasserstein between probability measures over images that are based on the convolution operators. We derive \emph{convolution sliced Wasserstein} (CSW) and its variants via incorporating stride, dilation, and non-linear activation function into the convolution operators. We investigate the metricity of CSW as well as its sample complexity, its computational complexity, and its connection to conventional sliced Wasserstein distances. Finally, we demonstrate the favorable performance of CSW over the conventional sliced Wasserstein in comparing probability measures over images and in training deep generative modeling on images.
BibTeX
@inproceedings{nguyen2022revisiting,
  author = {Khai Nguyen and Nhat Ho},
  title = {{Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=O4Q39aQFz0Y},
}
Figure