SetGANs: Enforcing Distributional Accuracy in Generative Adversarial Networks

Alessandro Ferrero, Shireen Y. Elhabian, Ross T. Whitaker

Abstract

Generative adversarial networks (GANs) have proven effective in modeling distributions of high-dimensional data. However, their training instability is a well-known hindrance to convergence, which results in practical challenges in their applications to novel data. Furthermore, even when convergence is reached, GANs can be affected by mode collapse, a phenomenon for which the generator learns to model only a small part of the target distribution, disregarding the vast majority of the data manifold or distribution. This paper addresses these challenges by introducing SetGAN, an adversarial architecture that processes sets of generated and real samples, and discriminates between the origins of these sets (i.e., training versus generated data) in a flexible, permutation invariant manner. We also propose a new metric to quantitatively evaluate GANs that does not require previous knowledge of the application, apart from the data itself. Using the new metric, in conjunction with the state-of-the-art evaluation methods, we show that the proposed architecture, when compared with GAN variants stemming from similar strategies, produces more accurate models of the input data in a way that is also less sensitive to hyperparameter settings.

Introduction

Generative Adversarial Networks (GANs) are powerful tools for modeling high-dimensional data distributions, yet they are notoriously difficult to train. Two primary obstacles—training instability and mode collapse—often prevent these models from capturing the full diversity of a target distribution. While previous attempts to stabilize training involved showing the discriminator multiple samples at once (e.g., Minibatch Discrimination or PacGAN), these methods lacked permutation invariance. This forced the networks to waste capacity learning that the order of samples in a batch should not affect the outcome. SetGAN addresses this by introducing a discriminator architecture that is mathematically guaranteed to be permutation invariant.

Methodology

SetGAN modifies the standard GAN objective by comparing sets of real and generated samples. The core innovation lies in the discriminator's three-part architecture, which ensures that the order of samples in a set does not change the classification output.

The SetGAN architecture consists of a Feature Subnetwork ($D_f$), a Pairing Subnetwork ($D_g$), and a Classifier Subnetwork ($D_h$) that utilizes differentiable soft histograms to achieve permutation invariance.
  1. Feature Subnetwork ($D_f$): Maps individual samples from a set into a latent feature space.
  2. Pairing Subnetwork ($D_g$): Processes pairs of these features to learn the underlying relationships between samples.
  3. Classifier Subnetwork ($D_h$): Aggregates these relationships using differentiable soft histograms. This creates a statistical summary of the set that remains identical regardless of input order, effectively smoothing the gradients and preventing the generator from falling into mode-collapse traps early in training.

Datasets and Evaluation

The authors evaluated SetGAN across three diverse benchmarks:

  • Stacked MNIST: A high-dimensional variant of MNIST designed to test a model's ability to capture numerous modes.
  • Galaxy10: A challenging dataset of ~22,000 astronomical images characterized by extreme class imbalance.
  • CelebA: A standard benchmark for high-quality face generation and hyperparameter sensitivity testing.

Results

SetGAN consistently outperformed standard GANs and existing packing methods like Minibatch Discrimination (MD) and PacGAN. On the Stacked MNIST benchmark, SetGAN5 achieved the lowest reverse KL-divergence (0.062), indicating superior mode coverage. In high-quality image generation on CelebA, it reached a state-of-the-art FID score of 18.92.

Qualitative comparison of generated samples on the Galaxy10 dataset. SetGAN5 demonstrates superior diversity and realism in galaxy morphology compared to DCGAN, MD, and PacGAN5.

Beyond image quality, SetGAN demonstrated remarkable robustness to hyperparameter tuning. While competing models showed volatile performance depending on learning rates or update ratios, SetGAN’s performance remained stable, significantly reducing the manual effort required for model optimization.

Conclusion

By embedding permutation invariance directly into the discriminator's architecture, SetGAN provides a robust solution to the common pitfalls of GAN training. The use of set-based summarization via soft histograms allows for cleaner training signals and better distributional accuracy. Future work may explore the scalability of this set-based approach to ultra-high-resolution imagery and its compatibility with other stabilization techniques like spectral normalization.

Summary updated

Abstract

Generative adversarial networks (GANs) have proven effective in modeling distributions of high-dimensional data. However, their training instability is a well-known hindrance to convergence, which results in practical challenges in their applications to novel data. Furthermore, even when convergence is reached, GANs can be affected by mode collapse, a phenomenon for which the generator learns to model only a small part of the target distribution, disregarding the vast majority of the data manifold or distribution. This paper addresses these challenges by introducing SetGAN, an adversarial architecture that processes sets of generated and real samples, and discriminates between the origins of these sets (i.e., training versus generated data) in a flexible, permutation invariant manner. We also propose a new metric to quantitatively evaluate GANs that does not require previous knowledge of the application, apart from the data itself. Using the new metric, in conjunction with the state-of-the-art evaluation methods, we show that the proposed architecture, when compared with GAN variants stemming from similar strategies, produces more accurate models of the input data in a way that is also less sensitive to hyperparameter settings.
BibTeX
@misc{ferrero2019setgans,
  author = {Alessandro Ferrero and Shireen Y. Elhabian and Ross T. Whitaker},
  title = {{SetGANs: Enforcing Distributional Accuracy in Generative Adversarial Networks}},
  year = {2019},
  eprint = {1907.00109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.00109},
}

Loading related content...

Figure