Generative modeling has long faced a "trilemma": the difficulty of simultaneously achieving high image quality, high sample diversity, and fast sampling speed. Generative Adversarial Networks (GANs) excel at speed but often suffer from mode collapse, limiting diversity. Conversely, Diffusion Models (DMs) provide exceptional quality and diversity but are computationally expensive due to their iterative sampling process.
Presented at AAAI-24, Consistency-GAN proposes a novel framework to bridge this gap. By integrating a "Consistency Mapping Module" into the GAN training pipeline, the authors achieve the diversity of diffusion models with the efficiency of GANs.
The Consistency-GAN Architecture
The core innovation of Consistency-GAN is the replacement of traditional, slow diffusion chains with a module based on Consistency Models. This allows for rapid mapping between noise and data distributions.
The framework consists of three primary components:
- Consistency Mapping Module: Unlike standard diffusion that requires many steps, this module maps any point on a diffusion trajectory back to the original data $x_0$ in very few steps (e.g., $NFE=5$). It facilitates efficient "instance noise" injection.
- Instance Noise Injection: By injecting noise into both real and generated samples, the model smooths the underlying data manifold. This prevents the discriminator from becoming overly dominant too quickly, a common cause of training instability and mode collapse in GANs.
- Step-Dependent Discriminator: The discriminator $D(y, n)$ is conditioned on the noise step $n$, allowing it to adapt its evaluation based on the level of perturbation applied to the input images.
Performance and Efficiency
Consistency-GAN was evaluated on CIFAR-10, STL-10, and CelebA datasets. The results demonstrate significant improvements over existing hybrid models like Diffusion-GAN.
Key Results:
- Superior Image Quality: On CIFAR-10, Consistency-GAN achieved a state-of-the-art Fréchet Inception Distance (FID) of 3.16, outperforming Diffusion-GAN (4.92) and BigGAN (14.7).
- Enhanced Diversity: The model reached a Recall score of 0.47, indicating high sample diversity and successful mitigation of mode collapse.
- Rapid Convergence: As shown in Figure 6, Consistency-GAN reaches a high-quality FID of 5 in just 4 hours (approx. 3,000 iterations), whereas Diffusion-GAN requires over 60 hours to reach the same performance level.
- Sampling Speed: The use of consistency mapping reduces the Number of Function Evaluations (NFE) to just 5 steps, making it roughly 6x faster than previous diffusion-hybrid approaches.
Conclusion
Consistency-GAN effectively addresses the generative trilemma by leveraging the strengths of consistency models within an adversarial framework. By smoothing the data manifold through efficient noise injection, it stabilizes GAN training and produces high-fidelity, diverse images at a fraction of the computational cost of traditional diffusion models. Future work may extend this framework to video and 3D synthesis.
Summary updated