A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rules

Sehwan Kim, Qifan Song, Faming LiangarXiv 2023 · DOI · arXiv

Introduction

Generative Adversarial Networks (GANs) have emerged as a cornerstone of modern generative modeling, yet they remain notoriously difficult to train. The most significant hurdle is mode collapse, a phenomenon where the generator learns to produce only a limited subset of the data distribution, effectively ignoring the diversity of the target dataset. This research introduces a novel framework—the Empirical Bayes-like GAN (EBGAN)—which reframes GAN training through the lens of statistical randomized decision theory to provide a more robust and theoretically grounded approach to generative modeling.

The Limitations of Pure Strategy Minimax

Standard GAN formulations typically treat the training process as a "pure strategy" minimax game. In this setup, the generator attempts to find a single optimal set of parameters to minimize the discriminator's ability to distinguish real from fake samples. However, because the discriminator evaluates samples on an individual basis, the generator often falls into sub-optimal local traps. Without a mechanism to force the generator to account for the entire population density, it naturally gravitates toward "safe" modes, leading to the loss of diversity.

The EBGAN Framework: Randomized Decision Rules

The authors propose shifting from a pure strategy to a Mixed Strategy Minimax Game. This approach treats the generator not as a single fixed point, but as a distribution of generators ($\pi_g$). This paradigm shift is supported by several key methodological innovations:

  1. Empirical Bayes-like Training: The framework treats the discriminator as a hyper-parameter within the generator's posterior distribution. At the Nash equilibrium, while the discriminator converges to a fixed point, the generator converges to a posterior distribution that captures the full complexity of the data.
  2. Stochastic Gradient MCMC: To navigate the generator's parameter space, the authors employ Momentum Stochastic Gradient Langevin Dynamics (MSGLD). Unlike standard gradient descent, which can easily stall in local optima, MSGLD allows the model to simulate and sample from the generator's posterior. This effectively "assembles" a mixture of generators that can cover all modes of the data distribution.
  3. KL-Divergence Prior: A novel prior based on Kullback-Leibler (KL) divergence is introduced. This forces the generated samples to match the real data at the population density level, providing a mathematical safeguard against mode collapse.

Experimental Validation

The EBGAN framework was evaluated across a diverse array of benchmarks, ranging from synthetic mixtures to complex biological datasets.

Mode Recovery and Image Quality

In synthetic tests involving a 10-component Gaussian mixture—a classic failure case for standard and even Lipschitz GANs—EBGAN successfully recovered all 10 components perfectly. On the Fashion-MNIST dataset, the model demonstrated superior performance in both image quality and diversity, achieving higher Inception Scores (IS) and lower Maximum Mean Discrepancy (MMD) than competing models like Bayesian GAN and ProbGAN.

High-Dimensional Clustering

EBGAN's ability to capture population density makes it an effective tool for nonparametric clustering. On the MNIST digit dataset, EBGAN achieved a Purity of 0.88 and an Adjusted Rand Index (ARI) of 0.77, significantly outperforming Cluster-GAN (Purity 0.72, ARI 0.56). It also successfully resolved the "Two-Circle" problem, a complex concentric ring structure that often defeats traditional clustering algorithms.

Biological Discovery

To demonstrate its utility in scientific research, the authors applied EBGAN to the Cancer Cell Line Encyclopedia (CCLE) dataset. The model was used to identify drug-sensitive genetic mutations. EBGAN successfully identified key mutations (such as MYC and CRKL) that were validated by biological literature but missed by previous GAN-based statistical tests, highlighting its potential for high-stakes scientific discovery.

Conclusion and Future Directions

By reframing GAN training as a mixed-strategy Nash equilibrium, this research provides a robust solution to the persistent problem of mode collapse. While the use of MCMC-based sampling introduces some computational overhead and requires careful hyper-parameter tuning (such as the "temperature" of the posterior), the theoretical guarantees and empirical results suggest a powerful new direction for generative modeling. Future work may explore the integration of sparse generators using spike-and-slab priors and the application of this framework to a broader class of risk functions.

Summary updated

Abstract

The Generative Adversarial Network (GAN) was recently introduced in the literature as a novel machine learning method for training generative models. It has many applications in statistics such as nonparametric clustering and nonparametric conditional independence tests. However, training the GAN is notoriously difficult due to the issue of mode collapse, which refers to the lack of diversity among generated data. In this paper, we identify the reasons why the GAN suffers from this issue, and to address it, we propose a new formulation for the GAN based on randomized decision rules. In the new formulation, the discriminator converges to a fixed point while the generator converges to a distribution at the Nash equilibrium. We propose to train the GAN by an empirical Bayes-like method by treating the discriminator as a hyper-parameter of the posterior distribution of the generator. Specifically, we simulate generators from its posterior distribution conditioned on the discriminator using a stochastic gradient Markov chain Monte Carlo (MCMC) algorithm, and update the discriminator using stochastic gradient descent along with simulations of the generators. We establish convergence of the proposed method to the Nash equilibrium. Apart from image generation, we apply the proposed method to nonparametric clustering and nonparametric conditional independence tests. A portion of the numerical results is presented in the supplementary material.
BibTeX
@misc{kim2023new,
  author = {Sehwan Kim and Qifan Song and Faming Liang},
  title = {{A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rules}},
  year = {2023},
  eprint = {2306.13641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.13641},
}
Figure