Abstract
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little diversity, even when trained on diverse datasets. This phenomenon, known as mode collapse, has been the main focus of several recent advances in GANs. Yet there is little understanding of why mode collapse happens and why existing approaches are able to mitigate mode collapse. We propose a principled approach to handling mode collapse, which we call packing. The main idea is to modify the discriminator to make decisions based on multiple samples from the same class, either real or artificially generated. We borrow analysis tools from binary hypothesis testing—in particular the seminal result of Blackwell [Bla53]—to prove a fundamental connection between packing and mode collapse. We show that packing naturally penalizes generators with mode collapse, thereby favoring generator distributions with less mode collapse during the training process. Numerical experiments on benchmark datasets suggests that packing provides significant improvements in practice as well.
Introduction
Generative Adversarial Networks (GANs) have revolutionized generative modeling, yet they suffer from a persistent flaw: mode collapse. This occurs when a generator learns to produce a very limited variety of outputs—for example, a model trained on all ten digits might only ever generate a high-quality "7." While researchers have proposed various heuristics to fix this, such as modifying loss functions or adding unrolling steps, we have lacked a rigorous theoretical understanding of why these fixes work or why mode collapse happens in the first place.
PacGAN introduces a simple yet mathematically principled modification called "packing" to solve this problem. By forcing the discriminator to analyze multiple samples simultaneously, the model naturally penalizes a lack of diversity, enabling generators to capture significantly more modes of a data distribution without complex hyperparameter tuning.
Methods: The Packing Framework
The core idea of PacGAN (Packed GAN) is "packing": instead of the discriminator looking at one sample at a time, it evaluates a group of $m$ samples.
The "Mother Architecture"
PacGAN acts as a wrapper for existing models. To implement it, one takes an existing GAN (the "mother") and modifies only the discriminator’s input layer to accept $m$ concatenated samples. The discriminator $D(X_1, \dots, X_m)$ must then decide if the entire group came from the real dataset or the generator. This requires minimal overhead, as all other hyperparameters—learning rate, batch size, and optimizer—remain identical to the original model.
Theoretical Foundation
The authors use tools from binary hypothesis testing to show that as the packing degree $m$ increases, the statistical "distance" between a diverse real distribution and a collapsed fake distribution grows much faster than the distance between two diverse distributions. By seeing multiple samples at once, "lack of diversity" becomes a visible feature that the discriminator can explicitly penalize.
Experimental Evaluation
The paper evaluates PacGAN across synthetic and real-world benchmarks:
- 2D-ring and 2D-grid: Synthetic mixtures of Gaussians where the number of modes is known exactly (8 and 25, respectively).
- Stacked MNIST: A dataset where three MNIST digits are stacked into color channels, creating 1,000 distinct modes to test extreme mode collapse.
- CelebA: A large-scale dataset of celebrity faces used to test diversity in high-dimensional image generation.
Key Results
PacGAN achieved significant improvements in diversity across all tests without the training instability often seen in other techniques.
Performance Highlights
- Stacked MNIST Victory: While a standard DCGAN captured only about 99 out of 1,000 modes, PacDCGAN2 captured all 1,000 modes consistently across trials.
- Synthetic Precision: On the 2D-grid (25 modes), PacGAN2 recovered all 25 modes, whereas the baseline GAN often recovered fewer than 20.
- Diversity vs. Capacity: A crucial insight is that PacGAN’s success isn't just due to increased discriminator parameters. Even when standard GANs were given more parameters to match PacGAN's capacity, they still suffered from mode collapse, while PacGAN did not.
Conclusion and Future Work
PacGAN demonstrates that mode collapse is essentially a failure of the discriminator to perform a sufficiently powerful hypothesis test. Packing provides the discriminator with the necessary "evidence" per decision to enforce diversity. While $m=2$ or $m=3$ provided massive gains in this study, future work may explore optimal packing degrees for different datasets or the use of permutation-invariant architectures like Deep Sets to further improve the processing of packed samples.
Summary updated