On Convergence in Wasserstein Distance and f-divergence Minimization Problems

Cheuk Ting Li, Jingwei Zhang, Farzan FarniaAISTATS 2024 · OpenReview

Introduction

Generative Adversarial Networks (GANs) have revolutionized generative modeling by framing the problem as a zero-sum game between a generator and a discriminator. Theoretically, this game minimizes a divergence measure between the real data distribution and the model's distribution. However, a persistent challenge is "mode collapse," where the generator focuses on a subset of the data distribution, ignoring other modes. This paper investigates how the choice of divergence measure—specifically $f$-divergences versus the 1-Wasserstein distance—influences the optimization landscape and the likelihood of mode collapse.

The Optimization Landscape: $f$-divergences vs. Wasserstein

The authors analyze the problem by fitting a mixture model to a target distribution. They categorize objectives into two main families:

  1. $f$-divergences: This includes the Jensen-Shannon (JS) divergence (used in vanilla GANs) and Reverse KL divergence. These measures are often "mode-seeking," prioritizing high-fidelity fitting of a single mode over broad coverage.
  2. Wasserstein Distance ($W_1$): Used in WGANs, this measure is based on optimal transport (the "Earth Mover's" distance).

The core finding is that $f$-divergences create optimization landscapes riddled with suboptimal local minima. Once a generator captures one mode effectively, moving toward a second mode initially increases the loss, creating a "valley" that gradient-based methods cannot cross. In contrast, the 1-Wasserstein distance possesses a "cancellation property," allowing the optimization to ignore correctly matched modes and focus purely on missing ones.

Theoretical Guarantees

The paper formalizes these observations through two key theorems:

  • Theorem 1 (Wasserstein Success): For well-separated data modes, gradient descent on the $W_1$ distance converges to the global optimum (capturing all modes) from a random initialization with high probability.
  • Theorem 2 ($f$-divergence Failure): For strongly mode-seeking $f$-divergences, there is no continuous path from a mode-collapsed state to a full-coverage state without increasing the loss, effectively trapping the model in a local minimum.

Synthetic Validation

To visualize these landscapes, the authors used a 5-mode Gaussian Mixture Model (GMM) to compare how different distances handle mode recovery.

Comparison of divergence measures on a 5-mode GMM. While JS and KL divergences remain trapped in a single mode, the Wasserstein distance ($W_1$) and hybrid measures successfully capture the full distribution.

As shown in Figure 1, the "Phase 1 Trained Generator" starts collapsed into a single central mode. When optimization continues using JS or KL divergence, the model remains stuck. However, $W_1$ and the $W_1$-JS Hybrid distance allow the generated data (orange) to spread across the real data modes (blue), eventually capturing the full geometry of the distribution.

Empirical Results on Image Datasets

The authors extended their analysis to real-world image data using bimodal datasets (e.g., mixing CelebA faces with CIFAR-10 ships, or color vs. grayscale versions of the same dataset). In "Phase 2" experiments, where a GAN was pre-trained on one mode and then introduced to two, JS-GANs consistently failed to "escape" the initial mode, maintaining a 100% focus on the first category. WGANs, conversely, reached a balanced ~50/50 distribution.

Generated samples from an LSUN Church dataset. While WGANs capture greater diversity across modes, they can sometimes produce noisier samples compared to the sharper but less diverse outputs of JS-GANs.

The trade-off for this diversity is often image quality. While JS-divergence is prone to collapse, the samples it does produce are often sharper. WGANs capture more modes but can result in "noisier" images that blur the boundaries between modes, as seen in the church generations in Figure 6. This suggests a bittersweet reality: the very property that allows $W_1$ to find new modes can lead to less precise local fitting.

Conclusion

The study provides a rigorous explanation for the empirical stability of WGANs. The $W_1$ distance's ability to avoid local optima makes it superior for capturing multi-modal distributions. However, the quality gap suggests that hybrid divergences—combining the diversity of Wasserstein with the local sharpness of $f$-divergences—represent the most promising path for future generative modeling research.

Summary updated

Abstract

The zero-sum game in generative adversarial networks (GANs) for learning the distribution of observed data is known to reduce to the minimization of a divergence measure between the underlying and generative models. However, the current theoretical understanding of the role of the target divergence in the characteristics of GANs’ generated samples remains largely inadequate. In this work, we aim to analyze the influence of the divergence measure on the local optima and convergence properties of divergence minimization problems in learning a multi-modal data distribution. We show a mode-seeking f-divergence, e.g. the Jensen-Shannon (JS) divergence in the vanilla GAN, could lead to poor locally optimal solutions missing some underlying modes. On the other hand, we demonstrate that the optimization landscape of 1-Wasserstein distance in Wasserstein GANs does not suffer from such suboptimal local minima. Furthermore, we prove that a randomly-initialized gradient-based optimization of the Wasserstein distance will, with high probability, capture all the existing modes. We present numerical results on standard image datasets, revealing the success of Wasserstein GANs compared to JS-GANs in avoiding suboptimal local optima under a mixture model.
BibTeX
@inproceedings{li2024convergence,
  author = {Cheuk Ting Li and Jingwei Zhang and Farzan Farnia},
  title = {{On Convergence in Wasserstein Distance and f-divergence Minimization Problems}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=1fmB1gK0te},
}
Figure