Introduction
Generative Adversarial Networks (GANs) have become a cornerstone of modern generative modeling, yet they are frequently hampered by "mode collapse." This phenomenon occurs when a generator learns to produce a limited subset of high-quality samples while ignoring significant portions of the target distribution's diversity. While empirical fixes abound, theoretical explanations have often suggested a fundamental trade-off: a discriminator powerful enough to prevent mode collapse will inevitably overfit due to high sample complexity, while a discriminator weak enough to generalize will fail to detect missing modes.
This paper challenges this dilemma by introducing the concept of restricted approximability. The authors demonstrate that by tailoring the discriminator's architecture to the generator's capacity, one can ensure that the GAN objective—the Integral Probability Metric (IPM)—faithfully approximates true statistical distances like Wasserstein or KL divergence. This alignment guarantees that if the training optimization succeeds, the GAN will capture the full diversity of the target distribution using only a polynomial number of samples.
The Generalization-Diversity Dilemma
Previous theoretical work (notably by Arora et al.) highlighted a depressing trade-off in GAN training. To truly prevent mode collapse, a discriminator needs to be extremely expressive (e.g., the class of all 1-Lipschitz functions). However, such broad classes require an exponentially large number of samples to generalize, leading to massive overfitting. In practice, researchers use neural networks as discriminators, which generalize well but compute a "weak" IPM. Historically, it was believed that a generator could easily fool these weak discriminators while still dropping modes.
This research asks a pivotal question: Is it possible to design a discriminator that is restricted enough to generalize with a polynomial number of samples, yet strong enough to detect and prevent mode collapse?
Methods: Restricted Approximability
The authors resolve the dilemma by designing discriminator classes specifically tailored to the family of distributions the generator can produce. Instead of requiring the discriminator to be universally powerful, it only needs to be strong enough to distinguish the target distribution from the specific types of distributions the generator is capable of outputting.
By mathematically constructing these specific discriminator families, the authors prove that the resulting IPM "sandwiches" true statistical distances. They provide constructions for several key classes:
- Gaussians: A simple 1-layer ReLU neural network is sufficient to bound the distance.
- Exponential Families: Linear combinations of sufficient statistics serve as the optimal discriminator.
- Invertible Neural Networks (INNs): A discriminator network with exactly one more layer than the generator network can provably approximate the KL divergence.
- Injective Neural Networks (Manifold Data): Since real-world data often lies on low-dimensional manifolds where standard metrics like KL divergence can become infinite, the authors designed a novel smoothed IPM variant that provably approximates the Wasserstein distance.
Empirical Validation
The authors validated their theoretical framework using synthetic datasets where true statistical distances can be precisely calculated:
- Geometric Manifolds: Using "Swiss Roll" and "Unit Circle" distributions, they found that the neural network IPM strongly correlates with the true Wasserstein distance throughout the training process.
- High-Dimensional INNs: For data generated by known invertible networks, the IPM showed a high correlation (0.73 to 0.75) with the true KL divergence. Interestingly, this correlation held even when using standard "vanilla" fully-connected discriminators, suggesting that the principle of restricted approximability may be naturally satisfied by common architectures.
Key Insights and Implications
- Tailored Discriminators are Essential: You do not need a universally powerful discriminator. You only need one that perfectly counters your specific generator's architecture.
- IPM as a Proxy: Under the condition of restricted approximability, the IPM (which GANs actually optimize) acts as a mathematically rigorous proxy for true statistical distances. If the IPM is small, diversity is guaranteed.
- Optimization vs. Statistics: The research suggests that mode collapse is likely an optimization failure rather than a statistical one. Because the theory shows a well-trained discriminator should prevent mode collapse, the lack of diversity seen in practice likely stems from the training algorithm failing to find the global minimum of the IPM.
Limitations and Future Work
While this paper provides a robust statistical guarantee for diversity, it does not solve the non-convex optimization problem inherent in GAN training. The proofs rely on the assumption that the training process can successfully minimize the expected IPM. Additionally, the specific architectures required for the proofs (e.g., a discriminator being exactly one layer deeper than the generator) are somewhat rigid. Future work is needed to bridge the gap between these theoretical constructions and the more flexible architectures used in large-scale production models, and to develop optimization algorithms that can more reliably reach the global minima identified by this theory.
Summary updated