On the Convergence and Robustness of Training GANs with Regularized Optimal Transport

Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, Jason D. Lee

Introduction

This paper introduces a mathematically principled approach to training Generative Adversarial Networks (GANs) by leveraging regularized Optimal Transport (OT). By smoothing the Wasserstein distance objective, the authors provide the first theoretical convergence guarantees to a stationary point for GANs. Furthermore, they demonstrate that this approach is both computationally efficient and highly robust in practice.

The Challenge of Training GANs

Training GANs is notoriously difficult and often relies on empirical heuristics. While the Wasserstein GAN (WGAN) improved training stability by using the Wasserstein distance as a metric between probability distributions, it introduced a new set of mathematical headaches. Specifically, the WGAN objective function is non-convex, non-smooth, and requires the discriminator to be a "1-Lipschitz" function—a constraint that is exceptionally hard to enforce strictly.

In practice, researchers rely on workarounds like weight clipping or gradient penalties to keep the discriminator in check. However, these techniques do not solve the underlying mathematical issue: the generator's objective remains non-smooth. This makes it nearly impossible to prove that the training will ever actually converge to a stable solution. This paper bridges the gap between the "black magic" of GAN training and rigorous optimization theory.

Introducing Smoothed WGAN (SWGAN)

To address these theoretical shortcomings, the authors propose Smoothed WGAN (SWGAN). Instead of using the standard Wasserstein distance, they employ a regularized version, which is closely related to the Sinkhorn distance.

The methodology relies on several key technical components:

  • Entropy/Norm-2 Regularization: By adding a regularization term to the Optimal Transport problem, the authors transform a "jagged," non-smooth objective into a "smooth" one that is differentiable everywhere.
  • Dual Formulation: They demonstrate that the dual of this regularized problem is an unconstrained, smooth, convex maximization problem. This allows a neural network discriminator to solve it much more reliably.
  • Oracle-based SGD: The authors propose a training loop where the discriminator is solved to "approximate optimality" ($\epsilon$-accuracy) before the generator takes a step.
  • Sinkhorn Loss: Regularization can introduce a "bias" that makes generated images look blurry. To fix this, the authors use a debiased Sinkhorn loss that maintains smoothness while accurately identifying the true data distribution.

Datasets and Evaluation

The authors evaluated their approach on two standard computer vision benchmarks:

  1. MNIST: Handwritten digits ($28 \times 28$ grayscale).
  2. CIFAR-10: Small color images ($32 \times 32$) across 10 categories.
Generated MNIST digits demonstrating the visual quality achieved by the proposed methods.

During evaluation, they experimented with different cost functions (such as L1 vs. Cosine distance) and latent representations, utilizing a Convolutional Neural Network to map images to a lower-dimensional space before calculating the OT distance.

Key Insights and Results

The transition to a smoothed objective yields several critical insights and performance improvements:

  • Smoothness is King: The fundamental breakthrough is that regularization fundamentally changes the geometry of the loss landscape. By making the objective $L$-smooth, the authors can finally apply standard optimization proofs (like those used for Stochastic Gradient Descent) to GANs.
  • Theoretical Guarantees: The paper provides a rigorous proof that their algorithm converges to a stationary point with a polynomial number of steps. This is a first for the GAN literature under such mild assumptions.
  • Discriminator Flexibility: The generator can still converge even if the discriminator isn't perfectly optimized. As long as the discriminator reaches a certain level of accuracy ($\epsilon$), the error in the generator's gradient remains bounded.
  • Faster Convergence and Robustness: On CIFAR-10, SWGAN achieved higher Inception Scores (a measure of image quality and diversity) faster than WGAN-GP (Gradient Penalty) and other OT-based methods. The method also proved flexible across different cost functions and architectures, showing less sensitivity to hyperparameter tuning than traditional WGANs.
MNIST digits generated by the baseline WGAN-GP method.

Limitations and Future Directions

While SWGAN offers a robust theoretical foundation, the authors note a few limitations. First, the theory assumes the discriminator can act as an "oracle" and be solved to $\epsilon$-accuracy; in practice, this depends heavily on the neural network's capacity and data complexity. Second, some of the proofs specifically target distributions with finite support (empirical samples), leaving the extension of full gradient error analysis to continuous distributions for future work. Finally, solving the discriminator to high accuracy at every step can introduce computational overhead, though the authors argue that the faster overall convergence compensates for this cost.

Summary updated

Abstract

Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10 images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.
BibTeX
@inproceedings{sanjabi2018convergence,
  author = {Maziar Sanjabi and Jimmy Ba and Meisam Razaviyayn and Jason D. Lee},
  title = {{On the Convergence and Robustness of Training GANs with Regularized Optimal Transport}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=6Iq4ipK04C},
}
Figure