Towards a Better Global Loss Landscape of GANs

Ruoyu Sun, Tiantian Fang, Alexander G. Schwing

TL;DR

Researchers have mathematically proven that standard Generative Adversarial Networks (GANs) are plagued by exponentially many "bad basins"—sub-optimal local minima that manifest as mode collapse. By switching to a Relativistic Pairing GAN (RpGAN), which couples generated and real samples in the loss function, these bad basins are eliminated. This leads to faster convergence and better performance, especially in resource-constrained (narrow) neural networks.

The Challenge: Mode Collapse and Bad Basins

Training GANs is notoriously difficult. While researchers have traditionally focused on convergence dynamics or statistical distances, this paper tackles the problem from the perspective of the global loss landscape.

The core challenge in GAN training is "mode collapse," where the generator produces only a limited variety of outputs, ignoring other valid modes in the training data. The authors argue that in standard GANs (like JS-GAN), mode collapse isn't just a fluke of the optimization algorithm—it is a fundamental, mathematical feature of the loss landscape itself. They identify "bad basins": large regions of the parameter space where the generator gets stuck in a sub-optimal state.

In standard Separable-GANs, fake samples (triangles) get stuck at local decision boundaries (red lines), ignoring other real data modes (circles) and causing mode collapse.

Methods: Separable vs. Relativistic Pairing GANs

The authors perform a global landscape analysis of the empirical loss of GANs, categorizing them into two distinct types:

  1. Separable-GANs (SepGAN): This category includes the original JS-GAN, Hinge-GAN, and Least-Squares GAN. In these models, the discriminator evaluates real and fake samples independently against a single decision boundary. The authors mathematically prove that for a dataset with $n$ points, standard Separable-GANs have $(n^n - n!)$ sub-optimal strict local minima. These are the "bad basins" that trap the generator.
  2. Relativistic Pairing GANs (RpGAN): These models "pair" a real sample with a fake sample in the loss function. Instead of asking "Is this sample real?", the discriminator effectively asks "Is this real sample more real than this fake one?"

The Power of Pairing

The breakthrough insight of the paper is that pairing samples breaks the locality of the discriminator's decision boundary.

In a standard GAN, the discriminator can "give up" on certain regions of the data space to focus on others, creating walls that the generator cannot cross. In RpGAN, the relative judgment forces the landscape to be "global-min-reachable." This means there is always a continuous, non-increasing path toward the optimal solution, effectively eliminating the bad basins.

By pairing real and fake samples, RpGAN allows the generator to bypass local boundaries and move samples toward all real data modes, eliminating bad basins.

Key Results and Empirical Evidence

The paper validates its theoretical claims across a wide range of benchmarks, including synthetic data, CIFAR-10, STL-10, and MNIST, yielding several compelling results:

  • Theoretical Guarantee: The authors formally prove that RpGAN has no bad basins and that any point in the landscape can reach the global minimum via a continuous path.
  • Faster Convergence: In two-cluster synthetic experiments, RpGAN escaped mode-collapse basins 3–4x faster than the standard JS-GAN.
  • Narrow Nets Benefit Most: One of the most interesting findings is that while wider neural networks can sometimes "smooth over" bad basins, narrow networks cannot. When using narrow, more efficient networks (e.g., a ResNet with only 28% of the standard parameters), RpGAN achieved an FID of 21.78 on CIFAR-10, significantly outperforming standard JS-GAN at the same width.
  • Robustness to Initialization: RpGAN demonstrated much lower sensitivity to bad initializations on MNIST compared to JS-GAN, which frequently got stuck in poor solutions.

Limitations and Future Work

While the findings are robust, the authors note a few limitations. First, Wasserstein GAN (W-GAN) was difficult to analyze globally due to its Lipschitz constraint. While they suspect W-GAN also avoids bad basins, a formal proof remains future work. Second, the parameter space proofs rely on a "path-keeping property" of the generator—a common but strong assumption that the neural network is expressive enough to represent the necessary paths. Finally, the paper focuses entirely on the landscape itself, leaving the complex interaction between this landscape and specific optimizers (like Adam) for future exploration.

Summary updated

Abstract

Understanding of GAN training is still very limited. One major challenge is its non-convex-non-concave min-max objective, which may lead to sub-optimal local minima. In this work, we perform a global landscape analysis of the empirical loss of GANs. We prove that a class of separable-GAN, including the original JS-GAN, has exponentially many bad basins which are perceived as mode-collapse. We also study the relativistic pairing GAN (RpGAN) loss which couples the generated samples and the true samples. We prove that RpGAN has no bad basins. Experiments on synthetic data show that the predicted bad basin can indeed appear in training. We also perform experiments to support our theory that RpGAN has a better landscape than separable-GAN. For instance, we empirically show that RpGAN performs better than separable-GAN with relatively narrow neural nets. The code is available at \url{https://github.com/AilsaF/RS-GAN}.
BibTeX
@inproceedings{sun2020towards,
  author = {Ruoyu Sun and Tiantian Fang and Alexander G. Schwing},
  title = {{Towards a Better Global Loss Landscape of GANs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=r3EZZBqbDl5},
}
Figure