Mode-Seeking Divergences: Theory and Applications to GANs

Cheuk Ting Li, Farzan FarniaAISTATS 2023 · OpenReview

The Mode-Seeking vs. Mode-Covering Dilemma

When training a Generative Adversarial Network (GAN) on a multi-modal dataset—such as one containing distinct clusters of cats and dogs—the generator faces a fundamental choice. It can either attempt to cover every mode in the training set, which often results in blurry "average" images in the low-density regions between modes, or it can focus on mastering a single mode perfectly, a phenomenon known as mode collapse.

These behaviors are formally categorized as mode-covering and mode-seeking. While practitioners have long observed that "Vanilla" GANs (using Jensen-Shannon divergence) produce sharper images but suffer from collapse, while Wasserstein GANs (WGANs) offer stability at the cost of realism, a rigorous theoretical explanation has been missing. This paper provides a unified framework to classify these behaviors and introduces the concept of "mode-seeking order" to quantify how divergences behave as data modes separate.

Theoretical Classification of Divergences

The authors analyze the scenario where a unimodal generator attempts to fit a multi-modal target distribution. They define four conditions (MS1-MS4) to classify $f$-divergences as weakly, strongly, or uniformly mode-seeking. For instance, the Neyman $\chi^2$ and Reverse-KL divergences are classified as uniformly mode-seeking, whereas the standard KL divergence is not mode-seeking at all.

Comparison of f-divergences fitting a bimodal distribution. Mode-seeking divergences (e.g., JS, Reverse-KL) focus on a single mode, while mode-covering ones (e.g., KL, Pearson) spread across both.

As shown in the figure above, when fitting a bimodal ground truth, mode-seeking divergences like JS and Reverse-KL "snap" to one of the modes, producing a sharp but incomplete representation. In contrast, mode-covering divergences like KL and Pearson $\chi^2$ attempt to bridge the gap, resulting in a wider distribution that covers both modes but places significant mass in the empty space between them.

The Wasserstein Limitation and Lipschitz Hybrids

A key insight of the paper is the "Wasserstein Surprise": the authors prove that for any dimension $d \geq 2$, the Wasserstein distance ($W_\rho$) is mathematically incapable of being mode-seeking. Instead, it pulls the generator toward the geometric median or mean of the data. This explains why WGANs often produce blurry samples in high-dimensional spaces.

To address this, the authors analyze Lipschitz GANs (which utilize Spectral Normalization or Gradient Penalty). They prove that these models actually minimize a hybrid divergence—a combination of an $f$-divergence and the 1-Wasserstein distance. In this setup, the Lipschitz constant ($K$) acts as a tunable knob: a higher $K$ increases mode-seeking behavior (sharper images), while a lower $K$ promotes mode-covering (greater diversity).

Empirical Evidence on GMMs

The theoretical framework was validated using synthetic Gaussian Mixture Models (GMMs). In experiments with 8-mode circular and 25-mode grid distributions, the differences between divergence types became starkly visible.

Empirical comparison on 8-mode and 25-mode GMMs. JS-GAN exhibits strong mode-seeking behavior (sharp points), while WGAN is mode-covering (diffuse clouds).

In the 25-mode grid experiment, mode-seeking GANs (like JS-GAN) successfully identified and centered on specific grid points. Conversely, the mode-covering WGAN placed the generator's mass in the empty spaces between the modes, resulting in the diffuse blue clouds seen in the visualization. Lipschitz-constrained versions (WGAN-GP) showed an improved balance, capturing the modes more distinctly than the standard WGAN while avoiding the total collapse seen in some JS-GAN runs.

Conclusion

By characterizing the mathematical properties that drive mode-seeking behavior, this work allows researchers to move beyond trial-and-error when selecting loss functions. The discovery that Lipschitz constraints create a hybrid divergence provides a theoretical justification for their success in modern GAN architectures, offering a principled way to navigate the trade-off between sample sharpness and distributional diversity.

Summary updated

Abstract

Generative adversarial networks (GANs) represent a game between two neural network machines designed to learn the distribution of data. It is commonly observed that different GAN formulations and divergence/distance measures used could lead to considerably different performance results, especially when the data distribution is multi-modal. In this work, we give a theoretical characterization of the mode-seeking behavior of general f-divergences and Wasserstein distances, and prove a performance guarantee for the setting where the underlying model is a mixture of multiple symmetric quasiconcave distributions. This can help us understand the trade-off between the quality and diversity of the trained GANs’ output samples. Our theoretical results show the mode-seeking nature of the Jensen-Shannon (JS) divergence over standard KL-divergence and Wasserstein distance measures. We subsequently demonstrate that a hybrid of JS-divergence and Wasserstein distance measures minimized by Lipschitz GANs mimics the mode-seeking behavior of the JS-divergence. We present numerical results showing the mode-seeking nature of the JS-divergence and its hybrid with the Wasserstein distance while highlighting the mode-covering properties of KL-divergence and Wasserstein distance measures. Our numerical experiments indicate the different behavior of several standard GAN formulations in application to benchmark Gaussian mixture and image datasets.
BibTeX
@inproceedings{li2023mode,
  author = {Cheuk Ting Li and Farzan Farnia},
  title = {{Mode-Seeking Divergences: Theory and Applications to GANs}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=XLH3qvnBdW},
}
Figure