TL;DR
This paper introduces CES-GAN, a novel framework that tackles the notorious mode collapse problem in Generative Adversarial Networks (GANs) by employing multiple discriminators with diverse cost functions. To mitigate training instability arising from unbalanced discriminators, they propose a gradient selection mechanism that intelligently picks the most beneficial gradients for the generator.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs) are powerful tools for generating realistic data, but they suffer from significant training instability and a phenomenon called "mode collapse." Mode collapse occurs when the generator fails to capture the full diversity of the real data distribution, producing only a limited subset of samples. This is often attributed to the difficulty of finding a continuous "transportation map" in the latent space, which GANs struggle to learn. While using multiple discriminators has been explored, their impact is limited if they share the same cost function. This paper aims to overcome mode collapse by leveraging the diverse gradients that arise from multiple discriminators with different cost functions, thereby enabling the generator to explore a wider range of data modes.
Methods
The core innovation is the Cost Ensemble with Gradient Selecting for GANs (CES-GAN) framework. It features:
- Multiple Discriminators with Different Cost Functions: Instead of a single discriminator, CES-GAN employs several, each trained with a distinct cost function. This diversity in gradients allows the generator to receive more varied signals, encouraging it to explore different modes of the data distribution.
- Gradient Selecting Mechanism: A critical component to address the instability introduced by multiple discriminators. The authors observe that a weak discriminator can provide misleading gradients, hindering generator training (akin to the "Cannikin Law" where the weakest part limits capacity). Their mechanism addresses this by:
- Designating a core discriminator whose gradients are always used to update the generator.
- Using auxiliary discriminators to provide supplementary gradients.
- Crucially, before updating the generator with an auxiliary discriminator's gradient, they calculate the Wasserstein distance between the real and generated distributions. Only if this distance is smaller than the current minimum is the gradient deemed "proper" and used for the update. This ensures that only gradients that demonstrably improve the generator's performance are selected.
- Auxiliary discriminators that consistently fail to provide useful gradients are discarded to save computational resources.
Datasets
The paper evaluates CES-GAN on several datasets:
- CIFAR-10: A standard dataset for image classification and generative model evaluation.
- MNIST: A dataset of handwritten digits.
- Stacked MNIST: A dataset where MNIST images are combined into a three-channel image, creating a larger number of modes.
- Toy Datasets (2D-ring and 2D-grid): Synthetic datasets with explicit distributions and known modes, used to precisely measure mode coverage and sample quality.
- CelebA and CelebA-HQ: Large-scale datasets of celebrity faces with attribute annotations, used for evaluating the generation of high-resolution images and attribute manipulation.
Key Insights and Ideas
- Diversity in Discriminator Cost Functions is Key: The paper strongly suggests that simply having more discriminators isn't enough; their differences in how they evaluate samples are what unlock the ability to explore more data modes.
- The "Cannikin Law" Applies to GANs: The analogy to the weakest link in a chain (or the shortest plank in a barrel) effectively highlights how a poorly performing discriminator can negatively impact the entire GAN training process.
- Wasserstein Distance as a Quality Gatekeeper: Using the Wasserstein distance as a criterion to select gradients is an elegant way to ensure that updates are genuinely improving the generator's ability to match the real data distribution, rather than just following a noisy signal.
- Intelligent Pruning of Discriminators: Discarding underperforming auxiliary discriminators is a practical optimization that prevents computational waste and maintains training efficiency.
Key Results
- Improved Mode Coverage: CES-GAN demonstrates a superior ability to capture modes compared to baseline GANs, as evidenced by experiments on toy datasets and Stacked MNIST. For instance, on the 2D-ring dataset, it achieved 24.8±0.2 modes out of 25.
- State-of-the-Art Performance: CES-GAN achieves competitive or superior results on various metrics. On CIFAR-10, it achieved the highest Inception Score (8.48) and lowest Fréchet Inception Distance (19.7) among compared methods.
- Effective Semi-Supervised Learning: In semi-supervised learning tasks on CIFAR-10, CES-GAN achieved a state-of-the-art test error of 9.36±0.20.
- High-Quality Image Generation: Qualitative results on CelebA and CelebA-HQ show the generation of realistic images with good continuity in interpolations.
Limitations and Future Work
- Computational Cost: While the gradient selection mechanism helps, training multiple discriminators inherently increases computational demands compared to a single-discriminator GAN.
- Hyperparameter Tuning: The choice of the core discriminator, the number of auxiliary discriminators, and their respective cost functions are important hyperparameters that may require careful tuning.
- Future Work: The paper suggests exploring different combinations of cost functions and further investigating the optimal number of discriminators. The authors also hint at the potential for this framework to be applied to other generative models beyond GANs.
Summary updated