Abstract: Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10 images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.
Introduction
Training Generative Adversarial Networks (GANs) is notoriously difficult, often likened to balancing a marble on a knife-edge. While the Wasserstein GAN (WGAN) formulation introduced stability by using the Wasserstein distance, it also brought a non-smooth objective function that is computationally grueling to optimize. Enforcing the 1-Lipschitz constraint on the discriminator typically relies on heuristics like weight clipping or gradient penalties, which can be unstable and lack strong theoretical backing.
Smoothing the Objective with Regularized Optimal Transport
The authors propose Smoothed WGAN (SWGAN), which transforms the jagged WGAN landscape into a smooth, differentiable surface. This is achieved by adding a strongly convex regularizer—such as KL divergence or an $L_2$ norm—to the Optimal Transport problem.
Key innovations of the SWGAN approach include:
- Differentiable Objective: Regularization turns the dual form of the Wasserstein distance into an unconstrained, smooth, concave maximization problem, making generator gradient calculations "computationally effortless."
- Sinkhorn Loss: To mitigate the bias introduced by regularization (which can "blur" the distance metric), the method utilizes the Sinkhorn loss. This allows for a larger, more stable regularization parameter $\lambda$ without sacrificing accuracy.
- Theoretical Convergence: The authors establish the first theoretical convergence guarantees to stationary points for this class of models, proving that solving the discriminator to only approximate optimality is sufficient for global progress.
Experimental Results
The researchers evaluated SWGAN on synthetic 2D Gaussians, MNIST, and CIFAR-10. In synthetic tests, SWGAN demonstrated superior robustness, perfectly recovering all 25 modes in settings where standard WGAN-GP failed.
On the MNIST dataset, SWGAN produced high-quality digits across various configurations. The experiments highlighted that applying cost functions in a lower-dimensional latent space (learned via CNN) often outperformed direct pixel-space optimization by avoiding poor local minima.
For the more complex CIFAR-10 dataset, SWGAN achieved competitive performance. Quantitative analysis using Inception scores shows that SWGAN consistently outperforms OT-GAN and WGAN-GP, reaching higher scores more efficiently. This confirms that the smooth approximation provides a more reliable gradient signal for the generator.
Conclusion
By prioritizing smoothness over exactness, SWGAN bridges the gap between the practical success of regularized Optimal Transport and the theoretical requirements for GAN convergence. The resulting algorithm is not only more robust and computationally efficient but also comes with the mathematical rigor previously lacking in the field. Future work may explore the scalability of these smoothed formulations to high-resolution image synthesis.
Summary updated