Introduction
Generative Adversarial Networks (GANs) have revolutionized generative modeling by framing the problem as a zero-sum game between a generator and a discriminator. Theoretically, this game minimizes a divergence measure between the real data distribution and the model's distribution. However, a persistent challenge is "mode collapse," where the generator focuses on a subset of the data distribution, ignoring other modes. This paper investigates how the choice of divergence measure—specifically $f$-divergences versus the 1-Wasserstein distance—influences the optimization landscape and the likelihood of mode collapse.
The Optimization Landscape: $f$-divergences vs. Wasserstein
The authors analyze the problem by fitting a mixture model to a target distribution. They categorize objectives into two main families:
- $f$-divergences: This includes the Jensen-Shannon (JS) divergence (used in vanilla GANs) and Reverse KL divergence. These measures are often "mode-seeking," prioritizing high-fidelity fitting of a single mode over broad coverage.
- Wasserstein Distance ($W_1$): Used in WGANs, this measure is based on optimal transport (the "Earth Mover's" distance).
The core finding is that $f$-divergences create optimization landscapes riddled with suboptimal local minima. Once a generator captures one mode effectively, moving toward a second mode initially increases the loss, creating a "valley" that gradient-based methods cannot cross. In contrast, the 1-Wasserstein distance possesses a "cancellation property," allowing the optimization to ignore correctly matched modes and focus purely on missing ones.
Theoretical Guarantees
The paper formalizes these observations through two key theorems:
- Theorem 1 (Wasserstein Success): For well-separated data modes, gradient descent on the $W_1$ distance converges to the global optimum (capturing all modes) from a random initialization with high probability.
- Theorem 2 ($f$-divergence Failure): For strongly mode-seeking $f$-divergences, there is no continuous path from a mode-collapsed state to a full-coverage state without increasing the loss, effectively trapping the model in a local minimum.
Synthetic Validation
To visualize these landscapes, the authors used a 5-mode Gaussian Mixture Model (GMM) to compare how different distances handle mode recovery.
As shown in Figure 1, the "Phase 1 Trained Generator" starts collapsed into a single central mode. When optimization continues using JS or KL divergence, the model remains stuck. However, $W_1$ and the $W_1$-JS Hybrid distance allow the generated data (orange) to spread across the real data modes (blue), eventually capturing the full geometry of the distribution.
Empirical Results on Image Datasets
The authors extended their analysis to real-world image data using bimodal datasets (e.g., mixing CelebA faces with CIFAR-10 ships, or color vs. grayscale versions of the same dataset). In "Phase 2" experiments, where a GAN was pre-trained on one mode and then introduced to two, JS-GANs consistently failed to "escape" the initial mode, maintaining a 100% focus on the first category. WGANs, conversely, reached a balanced ~50/50 distribution.
The trade-off for this diversity is often image quality. While JS-divergence is prone to collapse, the samples it does produce are often sharper. WGANs capture more modes but can result in "noisier" images that blur the boundaries between modes, as seen in the church generations in Figure 6. This suggests a bittersweet reality: the very property that allows $W_1$ to find new modes can lead to less precise local fitting.
Conclusion
The study provides a rigorous explanation for the empirical stability of WGANs. The $W_1$ distance's ability to avoid local optima makes it superior for capturing multi-modal distributions. However, the quality gap suggests that hybrid divergences—combining the diversity of Wasserstein with the local sharpness of $f$-divergences—represent the most promising path for future generative modeling research.
Summary updated