Amortized Projection Optimization for Sliced Wasserstein Generative Models

Khai Nguyen, Nhat Ho

Abstract

Finding informative projecting directions is a critical task in utilizing Sliced Wasserstein (SW) distances for generative modeling. Traditionally, identifying these directions requires an iterative optimization procedure that is computationally expensive, particularly in deep learning contexts where distances must be computed repeatedly across mini-batches. This nested optimization loop has historically limited the practical application of SW distances based on optimized projections. To address this, we introduce Amortized Sliced Wasserstein (A-SW), a framework that leverages "learning-to-optimize" techniques to predict informative directions for any given pair of mini-batch probability measures. By deriving linear, generalized linear, and non-linear amortized models, we create a family of novel mini-batch losses. Our results demonstrate that A-SW achieves superior performance in deep generative modeling across standard benchmarks while significantly reducing computational overhead.

Introduction

Generative modeling focuses on learning a model distribution ($\nu_\phi$) that matches a target data distribution ($\mu$). Optimal Transport (OT) distances, such as the Wasserstein distance, have emerged as powerful metrics for this task due to their ability to capture geometric structures. However, standard OT distances suffer from the "curse of dimensionality" and high computational complexity.

Sliced Wasserstein (SW) distances mitigate these issues by projecting high-dimensional distributions onto 1D lines, where the Wasserstein distance is computationally trivial. Despite this advantage, SW distances face a significant trade-off:

  1. Random Projections: Using many random directions requires a high number of projections ($L$) to achieve accuracy, which is memory-intensive.
  2. Max-Sliced Wasserstein (Max-SW): Instead of random sampling, one can search for the single best direction that maximizes the distance. While this yields higher-quality gradients, it introduces a nested optimization loop. For every update to the generator, a separate iterative optimization (often 10–100 steps) must be performed to find the optimal projection, making training prohibitively slow.

Amortized Sliced Wasserstein (A-SW)

To break this nested loop, we propose replacing the iterative search with Amortized Optimization. Rather than solving the maximization problem from scratch for every mini-batch, we train a parametric auxiliary function (an "amortized model") $f_\psi$ to predict the optimal projection direction directly from the data samples.

We introduce three distinct architectures for these amortized models:

  • Linear ($\mathcal{LA}$-SW): A streamlined model where the projection direction is computed as a weighted combination of the mini-batch samples.
  • Generalized Linear ($\mathcal{GA}$-SW): This variant applies non-linear mappings (such as Sigmoids) to the input mini-batches before combining them.
  • Non-linear ($\mathcal{NA}$-SW): This version utilizes a small neural network to map the combined mini-batch information to the projection space.

By training this auxiliary model alongside the generator in a minimax (adversarial-style) framework, the system learns to identify informative projections over time. This transforms an expensive iterative search into a single, efficient forward pass.

Experimental Results

We evaluated the A-SW framework on four standard image generation benchmarks: CIFAR10 (32x32), CelebA (64x64), STL10 (96x96), and CelebAHQ (128x128).

Generative Quality

Across all datasets, A-SW variants consistently outperformed standard SNGAN, SW, and Max-SW baselines in terms of FID scores and Inception Scores. For instance, on CIFAR10, the $\mathcal{LA}$-SW model achieved an FID of 13.21, a substantial improvement over Max-SW’s 31.33. This suggests that the amortized directions provide more meaningful gradients for the generator than either random or partially optimized directions.

Computational Efficiency

A-SW provides a significant speedup compared to Max-SW. In terms of iterations per second, $\mathcal{LA}$-SW performs nearly as fast as the baseline SNGAN and is much faster than Max-SW with 100 iterations ($T_2=100$). On the high-resolution CelebA-HQ dataset, $\mathcal{LA}$-SW processed approximately 10.19 iterations per second, whereas Max-SW ($T_2=100$) lagged at 8.13 iterations per second. Furthermore, A-SW avoids the memory bloat associated with using thousands of Monte Carlo projections ($L=10,000$), making it highly scalable.

Key Insights

  • The Power of Linearity: Surprisingly, the simplest version, $\mathcal{LA}$-SW, often performed as well as or better than more complex non-linear variants. This indicates that informative projection directions frequently lie within the span of the data samples themselves.
  • Shared Structure: The success of amortization proves that the optimal projection for a mini-batch is not random; it depends on the current state of the generator and the data. Amortization successfully leverages the shared structure across different mini-batches to predict these directions.
  • Faster Convergence: Because A-SW provides higher-quality directions than random sampling, generative models tend to converge in fewer epochs, reducing the total wall-clock training time.

Conclusion and Future Work

Amortized Sliced Wasserstein (A-SW) effectively bridges the gap between the efficiency of random projections and the accuracy of optimized projections. While we have demonstrated its efficacy, an "amortization gap" remains—the difference between the predicted projection and the true mathematical maximum. Future research will focus on understanding the theoretical properties of this gap and extending amortization techniques to other variants of Optimal Transport, such as Projected Robust Wasserstein or Sliced Mutual Information, which currently face similar computational bottlenecks.

Summary updated

Abstract

Seeking informative projecting directions has been an important task in utilizing sliced Wasserstein distance in applications. However, finding these directions usually requires an iterative optimization procedure over the space of projecting directions, which is computationally expensive. Moreover, the computational issue is even more severe in deep learning applications, where computing the distance between two mini-batch probability measures is repeated several times. This nested loop has been one of the main challenges that prevent the usage of sliced Wasserstein distances based on good projections in practice. To address this challenge, we propose to utilize the learning-to-optimize technique or amortized optimization to predict the informative direction of any given two mini-batch probability measures. To the best of our knowledge, this is the first work that bridges amortized optimization and sliced Wasserstein generative models. In particular, we derive linear amortized models, generalized linear amortized models, and non-linear amortized models which are corresponding to three types of novel mini-batch losses, named amortized sliced Wasserstein. We demonstrate the favorable performance of the proposed sliced losses in deep generative modeling on standard benchmark datasets.
BibTeX
@inproceedings{nguyen2022amortized,
  author = {Khai Nguyen and Nhat Ho},
  title = {{Amortized Projection Optimization for Sliced Wasserstein Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=ulz3BmFSEk},
}
Figure