Solving Approximate Wasserstein GANs to Stationarity

Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, Jason D. Lee

Abstract

Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10 images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.

Introduction

Generative Adversarial Networks (GANs) have revolutionized generative modeling but remain notoriously difficult to optimize. The Wasserstein GAN (WGAN) variant was introduced to improve stability by utilizing the Earth Mover's distance. However, the WGAN objective is inherently non-smooth and non-convex, presenting significant theoretical and practical challenges. Standard implementations rely on heuristics like weight clipping or gradient penalties to enforce Lipschitz constraints, but these lack rigorous convergence guarantees. This work addresses these gaps by introducing a smoothed formulation that allows for reliable gradient-based optimization.

Methodology: Smoothed WGAN (SWGAN)

The proposed Smoothed WGAN (SWGAN) leverages Regularized Optimal Transport (OT) to transform the problematic WGAN objective.

Regularization and Dual Formulation

By adding a strongly convex regularizer—such as KL divergence or an $L_2$ norm—to the primal optimal transport problem, the authors derive a dual formulation that is an unconstrained, smooth, and concave maximization. This shift makes finding the optimal discriminator computationally efficient. Crucially, the authors prove that the resulting generator objective is $L$-Lipschitz smooth, enabling the use of standard first-order optimization methods like Stochastic Gradient Descent (SGD).

De-biasing with Sinkhorn Loss

Large regularization parameters, while beneficial for optimization stability, can introduce bias that leads to blurry generated images. To mitigate this, the authors employ the Sinkhorn loss. This de-biasing technique preserves the optimization advantages of high regularization while maintaining the sharpness and quality of the generated samples.

Optimization Algorithm

The paper introduces an oracle-based SGD algorithm. A key theoretical finding is that the discriminator only needs to be solved to "approximate optimality" at each step to ensure the generator reaches a stationary point in polynomial time.

Experimental Results

The SWGAN approach was evaluated across synthetic and real-world datasets, demonstrating both theoretical robustness and empirical performance.

Synthetic 2D Gaussians

In a grid of 25 Gaussians, SWGAN successfully recovered all modes. It demonstrated superior robustness compared to WGAN-GP, particularly when the latent code dimension was significantly reduced, a condition where standard methods often suffer from mode collapse.

MNIST and CIFAR-10

On standard benchmarks like MNIST and CIFAR-10, SWGAN produced high-quality images. The method proved to be computationally efficient, often converging faster than baseline architectures.

Generated MNIST samples using the SWGAN framework.
Additional samples of handwritten digits generated by the proposed model.

On the CIFAR-10 dataset, the Sinkhorn-based objective achieved competitive Inception Scores more rapidly than WGAN-GP, highlighting the efficiency of the smoothed objective.

Discussion and Key Insights

  • Smoothing as a Catalyst: The transition from a non-smooth to a smooth objective is the primary driver for reliable gradient information and rigorous convergence analysis.
  • Discriminator Accuracy: The proximity to a global solution is tied to the accuracy of the discriminator at each step. While an "oracle" is assumed theoretically, practical neural network approximations are sufficient for high-quality results.
  • Latent Space Optimization: Applying cost functions in a learned latent space rather than raw pixel space helps the model navigate high-dimensional landscapes more effectively.

Conclusion and Future Work

This work provides the first global convergence guarantee to a stationary solution for GANs under mild assumptions. While the gap between theoretical "oracles" and practical neural network discriminators remains, the empirical success of SWGAN on MNIST and CIFAR-10 validates the approach. Future research may explore more complex cost functions and further bridge the gap between functional-space theory and parametric implementations.

Summary updated

Abstract

Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.
BibTeX
@inproceedings{sanjabi2018solving,
  author = {Maziar Sanjabi and Jimmy Ba and Meisam Razaviyayn and Jason D. Lee},
  title = {{Solving Approximate Wasserstein GANs to Stationarity}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=6Iq4ipK04C},
}
Figure