The Mode-Seeking vs. Mode-Covering Dilemma
When training a Generative Adversarial Network (GAN) on a multi-modal dataset—such as one containing distinct clusters of cats and dogs—the generator faces a fundamental choice. It can either attempt to cover every mode in the training set, which often results in blurry "average" images in the low-density regions between modes, or it can focus on mastering a single mode perfectly, a phenomenon known as mode collapse.
These behaviors are formally categorized as mode-covering and mode-seeking. While practitioners have long observed that "Vanilla" GANs (using Jensen-Shannon divergence) produce sharper images but suffer from collapse, while Wasserstein GANs (WGANs) offer stability at the cost of realism, a rigorous theoretical explanation has been missing. This paper provides a unified framework to classify these behaviors and introduces the concept of "mode-seeking order" to quantify how divergences behave as data modes separate.
Theoretical Classification of Divergences
The authors analyze the scenario where a unimodal generator attempts to fit a multi-modal target distribution. They define four conditions (MS1-MS4) to classify $f$-divergences as weakly, strongly, or uniformly mode-seeking. For instance, the Neyman $\chi^2$ and Reverse-KL divergences are classified as uniformly mode-seeking, whereas the standard KL divergence is not mode-seeking at all.
As shown in the figure above, when fitting a bimodal ground truth, mode-seeking divergences like JS and Reverse-KL "snap" to one of the modes, producing a sharp but incomplete representation. In contrast, mode-covering divergences like KL and Pearson $\chi^2$ attempt to bridge the gap, resulting in a wider distribution that covers both modes but places significant mass in the empty space between them.
The Wasserstein Limitation and Lipschitz Hybrids
A key insight of the paper is the "Wasserstein Surprise": the authors prove that for any dimension $d \geq 2$, the Wasserstein distance ($W_\rho$) is mathematically incapable of being mode-seeking. Instead, it pulls the generator toward the geometric median or mean of the data. This explains why WGANs often produce blurry samples in high-dimensional spaces.
To address this, the authors analyze Lipschitz GANs (which utilize Spectral Normalization or Gradient Penalty). They prove that these models actually minimize a hybrid divergence—a combination of an $f$-divergence and the 1-Wasserstein distance. In this setup, the Lipschitz constant ($K$) acts as a tunable knob: a higher $K$ increases mode-seeking behavior (sharper images), while a lower $K$ promotes mode-covering (greater diversity).
Empirical Evidence on GMMs
The theoretical framework was validated using synthetic Gaussian Mixture Models (GMMs). In experiments with 8-mode circular and 25-mode grid distributions, the differences between divergence types became starkly visible.
In the 25-mode grid experiment, mode-seeking GANs (like JS-GAN) successfully identified and centered on specific grid points. Conversely, the mode-covering WGAN placed the generator's mass in the empty spaces between the modes, resulting in the diffuse blue clouds seen in the visualization. Lipschitz-constrained versions (WGAN-GP) showed an improved balance, capturing the modes more distinctly than the standard WGAN while avoiding the total collapse seen in some JS-GAN runs.
Conclusion
By characterizing the mathematical properties that drive mode-seeking behavior, this work allows researchers to move beyond trial-and-error when selecting loss functions. The discovery that Lipschitz constraints create a hybrid divergence provides a theoretical justification for their success in modern GAN architectures, offering a principled way to navigate the trade-off between sample sharpness and distributional diversity.
Summary updated