Introduction
This paper introduces a mathematically principled approach to training Generative Adversarial Networks (GANs) by leveraging regularized Optimal Transport (OT). By smoothing the Wasserstein distance objective, the authors provide the first theoretical convergence guarantees to a stationary point for GANs. Furthermore, they demonstrate that this approach is both computationally efficient and highly robust in practice.
The Challenge of Training GANs
Training GANs is notoriously difficult and often relies on empirical heuristics. While the Wasserstein GAN (WGAN) improved training stability by using the Wasserstein distance as a metric between probability distributions, it introduced a new set of mathematical headaches. Specifically, the WGAN objective function is non-convex, non-smooth, and requires the discriminator to be a "1-Lipschitz" function—a constraint that is exceptionally hard to enforce strictly.
In practice, researchers rely on workarounds like weight clipping or gradient penalties to keep the discriminator in check. However, these techniques do not solve the underlying mathematical issue: the generator's objective remains non-smooth. This makes it nearly impossible to prove that the training will ever actually converge to a stable solution. This paper bridges the gap between the "black magic" of GAN training and rigorous optimization theory.
Introducing Smoothed WGAN (SWGAN)
To address these theoretical shortcomings, the authors propose Smoothed WGAN (SWGAN). Instead of using the standard Wasserstein distance, they employ a regularized version, which is closely related to the Sinkhorn distance.
The methodology relies on several key technical components:
- Entropy/Norm-2 Regularization: By adding a regularization term to the Optimal Transport problem, the authors transform a "jagged," non-smooth objective into a "smooth" one that is differentiable everywhere.
- Dual Formulation: They demonstrate that the dual of this regularized problem is an unconstrained, smooth, convex maximization problem. This allows a neural network discriminator to solve it much more reliably.
- Oracle-based SGD: The authors propose a training loop where the discriminator is solved to "approximate optimality" ($\epsilon$-accuracy) before the generator takes a step.
- Sinkhorn Loss: Regularization can introduce a "bias" that makes generated images look blurry. To fix this, the authors use a debiased Sinkhorn loss that maintains smoothness while accurately identifying the true data distribution.
Datasets and Evaluation
The authors evaluated their approach on two standard computer vision benchmarks:
- MNIST: Handwritten digits ($28 \times 28$ grayscale).
- CIFAR-10: Small color images ($32 \times 32$) across 10 categories.
During evaluation, they experimented with different cost functions (such as L1 vs. Cosine distance) and latent representations, utilizing a Convolutional Neural Network to map images to a lower-dimensional space before calculating the OT distance.
Key Insights and Results
The transition to a smoothed objective yields several critical insights and performance improvements:
- Smoothness is King: The fundamental breakthrough is that regularization fundamentally changes the geometry of the loss landscape. By making the objective $L$-smooth, the authors can finally apply standard optimization proofs (like those used for Stochastic Gradient Descent) to GANs.
- Theoretical Guarantees: The paper provides a rigorous proof that their algorithm converges to a stationary point with a polynomial number of steps. This is a first for the GAN literature under such mild assumptions.
- Discriminator Flexibility: The generator can still converge even if the discriminator isn't perfectly optimized. As long as the discriminator reaches a certain level of accuracy ($\epsilon$), the error in the generator's gradient remains bounded.
- Faster Convergence and Robustness: On CIFAR-10, SWGAN achieved higher Inception Scores (a measure of image quality and diversity) faster than WGAN-GP (Gradient Penalty) and other OT-based methods. The method also proved flexible across different cost functions and architectures, showing less sensitivity to hyperparameter tuning than traditional WGANs.
Limitations and Future Directions
While SWGAN offers a robust theoretical foundation, the authors note a few limitations. First, the theory assumes the discriminator can act as an "oracle" and be solved to $\epsilon$-accuracy; in practice, this depends heavily on the neural network's capacity and data complexity. Second, some of the proofs specifically target distributions with finite support (empirical samples), leaving the extension of full gradient error analysis to continuous distributions for future work. Finally, solving the discriminator to high accuracy at every step can introduce computational overhead, though the authors argue that the faster overall convergence compensates for this cost.
Summary updated