Abstract
Despite its success, generative adversarial networks (GANs) still suffer from mode collapse, i.e., the generator can only map latent variables to a partial set of modes in the target distribution. In this paper, we analyze and seek to regularize this issue with an independent and identically distributed (IID) sampling perspective and emphasize that holding the IID property referring to the target distribution for generation can naturally avoid mode collapse. This is based on the basic IID assumption for real data in machine learning. However, though the source samples {z} obey IID, the generations {G(z)} may not necessarily be IID sampling from the target distribution. Based on this observation, considering a necessary condition of IID generation, we propose a new loss to encourage the closeness between the inverse samples of real data and the Gaussian source in the latent space to regularize the generation to be IID from the target distribution. Experiments on both synthetic and real-world data show the effectiveness of our model.
Introduction
Generative Adversarial Networks (GANs) have revolutionized image synthesis, yet they remain plagued by mode collapse. This phenomenon occurs when a generator learns to produce only a narrow subset of the target distribution's modes, failing to capture the full diversity of the training data. While many existing approaches focus on improving the visual quality of individual samples, they often overlook the collective statistical properties of the generated set.
IID-GAN introduces a novel perspective by focusing on the Independent and Identically Distributed (IID) property. In standard machine learning, we assume training data is IID. If a generator perfectly captures the target distribution, then the inverse mapping of real data back into the latent space should also be IID and follow the original source distribution (typically a standard Gaussian). Most GANs fail this test: if you encode real data back into the latent space of a collapsed GAN, the resulting points will not follow the expected standard Gaussian distribution.
Methodology
The core innovation of IID-GAN is the Gaussian Consistency Loss ($L_{Gau}$), which enforces statistical regularization at the batch level. The framework consists of three main components:
- Adversarial Learning: A standard Generator ($G$) and Discriminator ($D$) pair.
- Cycle-Consistency: An encoder ($F$) is introduced to learn the inverse mapping $x \to z$. This ensures a bidirectional link where $G(F(x)) \approx x$ and $F(G(z)) \approx z$.
- Gaussian Consistency: The model calculates the mean and covariance of a batch of real data encoded into the latent space. It then uses the Wasserstein distance to minimize the discrepancy between these batch statistics and a standard Gaussian distribution ($\mu=0, \sigma=I$).
By forcing the entire batch of real data to map to a standard Gaussian in the latent space, the generator is mathematically incentivized to cover all modes of the target distribution to "fill out" the required Gaussian distribution in the latent space.
Experiments and Results
The authors evaluated IID-GAN across synthetic 2D datasets (Ring and Grid) and real-world benchmarks including MNIST, Stacked MNIST, CIFAR-10, and high-resolution datasets like CelebA and AFHQ Cat.
- Mode Coverage: On the Stacked MNIST benchmark, which contains 1,000 possible modes, IID-GAN captured 999.7 modes on average. This significantly outperforms vanilla GANs (392 modes) and specialized architectures like VEEGAN (761.8 modes).
- Stability and Scalability: IID-GAN demonstrated superior training stability. On CIFAR-10, while competing models like MSGAN saw their FID scores deteriorate over time, IID-GAN remained consistent. Furthermore, when integrated with state-of-the-art backbones like StyleGAN2-ada, IID-GAN further improved both FID and KID scores on complex datasets like AFHQ Cat and LSUN Church.
- Batch-Level Regularization: Unlike Variational Autoencoders (VAEs) which regularize samples individually, IID-GAN regularizes the distribution of a batch. This allows for more flexibility in individual samples while maintaining global diversity.
Conclusion
IID-GAN provides a theoretically grounded yet practical solution to mode collapse by leveraging the IID assumption of real data. By ensuring that the inverse mapping of real data matches the source distribution, the model naturally achieves higher diversity and stability. While the addition of an encoder introduces some computational overhead, the empirical gains in mode coverage and training consistency make it a compelling approach for robust generative modeling.
Summary updated