TL;DR
This paper introduces Diffusion-GAN, a novel framework that uses a diffusion process to inject instance noise into GAN training, significantly improving stability and data efficiency. By adaptively adjusting the diffusion process, Diffusion-GAN provides a robust and domain-agnostic augmentation method that outperforms existing state-of-the-art GANs in generating high-fidelity images.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs) are powerful for generating realistic images but notoriously difficult to train stably, often suffering from issues like non-convergence and mode collapse. A proposed solution is to inject "instance noise" into the discriminator's input to stabilize training. However, this technique has proven challenging to implement effectively in practice, particularly for high-dimensional data like images, as finding the right noise distribution is difficult. Existing methods that attempt to address this, like gradient penalties, can sometimes improve stability at the cost of generation quality. This paper aims to provide a practical and effective way to inject instance noise that stabilizes GAN training without sacrificing generation performance.
Methods
The core innovation of Diffusion-GAN is its use of a forward diffusion chain to generate instance noise. Instead of directly adding noise, the method applies a series of diffusion steps to both real and generated data. This process gradually adds Gaussian noise, effectively creating noisy versions of the data at various "diffusion timesteps."
Key components include:
- Adaptive Diffusion Process: The number of diffusion steps (and thus the amount of noise) is not fixed but adaptively adjusted during training. This is controlled by a metric that monitors discriminator overfitting, allowing the model to gradually increase the difficulty of the task for the discriminator.
- Timestep-Dependent Discriminator: The discriminator takes both the diffused data and the diffusion timestep as input, allowing it to learn from data at different noise levels.
- Differentiable Generator Update: The generator's gradients are backpropagated through the forward diffusion chain. This is enabled by the diffusion process being differentiable.
- Gaussian Mixture Noise: The instance noise is sampled from a Gaussian mixture distribution across all diffusion steps. This ensures that the generator receives consistent and helpful guidance from the discriminator across various noise levels.
This approach is described as "model- and domain-agnostic," meaning it can be applied to various GAN architectures and data types without requiring specific domain knowledge.
Datasets
The paper evaluates Diffusion-GAN on a wide range of datasets, including:
- Low-resolution: CIFAR-10 (32x32)
- Medium-resolution: STL-10 (64x64), CelebA (64x64)
- High-resolution: LSUN-Bedroom (256x256), LSUN-Church (256x256), AFHQ (Cat/Dog/Wild, 512x512), FFHQ (1024x1024)
They also conduct experiments on a 2D toy dataset (25-Gaussians) and use feature vectors from ProjectedGAN for domain-agnostic evaluation.
Key Insights and Ideas
- Diffusion as a Stable Noise Injector: The diffusion process naturally provides a smooth, controllable way to add noise, overcoming the practical difficulties of defining suitable noise distributions for instance noise injection in GANs.
- Adaptive Difficulty for Discriminator: By adaptively adjusting the diffusion steps, the model can present a progressively harder task to the discriminator, preventing it from overfitting too early and ensuring it provides useful learning signals to the generator.
- Non-Leaking Augmentation: The diffusion process is shown to be a "non-leaking" augmentation, meaning it doesn't introduce artifacts or biases that could harm generation quality, a problem sometimes seen with other augmentation techniques.
- Theoretical Guarantees: The paper provides theoretical analysis showing that Diffusion-GAN's objective function is continuous and differentiable, ensuring that the generator always receives useful gradients.
Key Results
Diffusion-GAN consistently improves the performance of various GAN baselines:
- Improved Fidelity and Diversity: Diffusion-GAN, when applied to models like StyleGAN2, ProjectedGAN, and InsGen, achieves state-of-the-art results on multiple benchmark datasets, outperforming baselines in both Fréchet Inception Distance (FID) for fidelity and Recall for diversity. For example, on CIFAR-10, Diffusion StyleGAN2 achieved an FID of 3.19 and Recall of 0.58, outperforming StyleGAN2 (FID 8.32, Recall 0.41).
- Data Efficiency: The method significantly boosts performance in data-limited scenarios. For instance, Diffusion InsGen on FFHQ with only 5k images achieved an FID of 16.43, outperforming InsGen (18.21).
- Domain Agnosticism: Applying Diffusion-GAN to feature vectors in ProjectedGAN also showed clear improvements in FID and Recall, demonstrating its applicability beyond image pixels.
- Training Stability: Experiments show that Diffusion-GAN leads to more stable training, with adaptive diffusion strategies converging faster and reaching lower final FID scores.
Limitations and Future Work
- Mixing Procedure Optimization: The paper notes that while the "priority mixing" strategy for sampling diffusion steps is intuitive, an ablation study suggests that "uniform mixing" might perform better on certain datasets. Optimizing this mixing procedure for different datasets is suggested as future work.
- Computational Cost: While generally comparable to baselines, the diffusion process itself adds some computational overhead. The paper mentions that for StyleGAN2, the added memory cost is negative and training time cost is negative, while inference time is zero, suggesting efficiency gains in some aspects.
- Further Investigation: The authors suggest that further investigation into optimizing the mixing procedure and exploring more satisfying possibilities for differentiable and invertible mapping functions could be beneficial.
Summary updated