Abstract
Generative adversarial networks (GANs) have proven effective in modeling distributions of high-dimensional data. However, their training instability is a well-known hindrance to convergence, which results in practical challenges in their applications to novel data. Furthermore, even when convergence is reached, GANs can be affected by mode collapse, a phenomenon for which the generator learns to model only a small part of the target distribution, disregarding the vast majority of the data manifold or distribution. This paper addresses these challenges by introducing SetGAN, an adversarial architecture that processes sets of generated and real samples, and discriminates between the origins of these sets (i.e., training versus generated data) in a flexible, permutation invariant manner. We also propose a new metric to quantitatively evaluate GANs that does not require previous knowledge of the application, apart from the data itself. Using the new metric, in conjunction with the state-of-the-art evaluation methods, we show that the proposed architecture, when compared with GAN variants stemming from similar strategies, produces more accurate models of the input data in a way that is also less sensitive to hyperparameter settings.
Introduction
Generative Adversarial Networks (GANs) are powerful tools for high-dimensional data modeling but are notoriously difficult to train due to two primary issues: training instability and mode collapse. Training instability often manifests as vanishing or exploding gradients, disrupting the equilibrium between the generator and discriminator. Mode collapse occurs when the generator learns to produce only a limited subset of the target distribution, failing to capture the full diversity of the training data.
While previous approaches like PacGAN (packing samples) or Minibatch Discrimination have attempted to mitigate these issues by allowing the discriminator to view multiple samples simultaneously, they often introduce sensitivity to sample ordering or deviate from the theoretical optimality of the original GAN framework. SetGAN addresses these limitations by employing a permutation-invariant architecture that evaluates sets of samples rather than individual ones.
Methodology: The SetGAN Architecture
The core innovation of SetGAN is its discriminator, which is designed to be permutation invariant—meaning the output is independent of the order of samples within a batch. This architectural constraint prevents the discriminator from wasting capacity on learning that order is irrelevant, a common pitfall in simpler packing methods.
As shown in Figure 1, the SetGAN discriminator consists of three specialized subnetworks:
- Feature Subnetwork ($D_f$): Maps individual samples in a set to latent feature representations.
- Pairing Subnetwork ($D_g$): Processes pairs of these feature vectors to learn inter-sample relationships.
- Classifier Subnetwork ($D_h$): Utilizes a differentiable "soft histogram" approximation to create a statistical summary of the features, which informs the final "real vs. fake" classification for the entire set.
This set-based approach ensures that gradients remain informative even during the early stages of mode collapse, preventing the overfitting cycles that typically degrade generator diversity.
Experimental Setup
SetGAN was evaluated across three distinct benchmarks:
- Stacked MNIST: A dataset where each sample contains three random digits (1,000 possible modes), specifically designed to test a model's ability to capture diversity.
- Galaxy10: Approximately 22,000 astronomical photographs with high class imbalance, testing the model's performance on rare data types.
- CelebA: A large-scale dataset of ~202,000 celebrity faces used to evaluate image quality and hyperparameter sensitivity.
Key Results and Discussion
SetGAN demonstrates superior performance in both diversity and image quality. On the Stacked MNIST benchmark, SetGAN5 (using sets of 5) achieved a Reversed KL-divergence of 0.062, significantly outperforming PacGAN5 (0.258) and standard GANs (0.081).
In terms of image quality, SetGAN achieved the lowest Fréchet Inception Distance (FID) across all tested datasets. On CelebA, it reached an FID of 18.92, compared to 25.51 for a standard DCGAN. Furthermore, SetGAN exhibits remarkable stability; its performance remains consistent across various learning rates and update ratios (GD-ratio), whereas standard GANs and PacGANs often suffer from training failure or gradient saturation under similar conditions.
Conclusion
SetGAN provides a robust framework for stabilizing GAN training and preventing mode collapse through permutation-invariant set processing. By utilizing differentiable soft histograms, the architecture maintains rich statistical summaries that guide the generator more effectively. While the pairing subnetwork introduces additional computational overhead, the gains in stability and diversity make SetGAN a compelling alternative for high-dimensional data modeling. Future work may explore scaling set sizes and integrating this framework into more advanced architectures like StyleGAN.
Summary updated