Diffusion-GAN: Training GANs with Diffusion

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, Mingyuan Zhou

TL;DR

This paper introduces Diffusion-GAN, a novel framework that uses a diffusion process to inject instance noise into GAN training, significantly improving stability and data efficiency. By adaptively adjusting the diffusion process, Diffusion-GAN provides a robust and domain-agnostic augmentation method that outperforms existing state-of-the-art GANs in generating high-fidelity images.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful for generating realistic images but notoriously difficult to train stably, often suffering from issues like non-convergence and mode collapse. A proposed solution is to inject "instance noise" into the discriminator's input to stabilize training. However, this technique has proven challenging to implement effectively in practice, particularly for high-dimensional data like images, as finding the right noise distribution is difficult. Existing methods that attempt to address this, like gradient penalties, can sometimes improve stability at the cost of generation quality. This paper aims to provide a practical and effective way to inject instance noise that stabilizes GAN training without sacrificing generation performance.

Methods

The core innovation of Diffusion-GAN is its use of a forward diffusion chain to generate instance noise. Instead of directly adding noise, the method applies a series of diffusion steps to both real and generated data. This process gradually adds Gaussian noise, effectively creating noisy versions of the data at various "diffusion timesteps."

Key components include:

  • Adaptive Diffusion Process: The number of diffusion steps (and thus the amount of noise) is not fixed but adaptively adjusted during training. This is controlled by a metric that monitors discriminator overfitting, allowing the model to gradually increase the difficulty of the task for the discriminator.
  • Timestep-Dependent Discriminator: The discriminator takes both the diffused data and the diffusion timestep as input, allowing it to learn from data at different noise levels.
  • Differentiable Generator Update: The generator's gradients are backpropagated through the forward diffusion chain. This is enabled by the diffusion process being differentiable.
  • Gaussian Mixture Noise: The instance noise is sampled from a Gaussian mixture distribution across all diffusion steps. This ensures that the generator receives consistent and helpful guidance from the discriminator across various noise levels.

This approach is described as "model- and domain-agnostic," meaning it can be applied to various GAN architectures and data types without requiring specific domain knowledge.

Datasets

The paper evaluates Diffusion-GAN on a wide range of datasets, including:

  • Low-resolution: CIFAR-10 (32x32)
  • Medium-resolution: STL-10 (64x64), CelebA (64x64)
  • High-resolution: LSUN-Bedroom (256x256), LSUN-Church (256x256), AFHQ (Cat/Dog/Wild, 512x512), FFHQ (1024x1024)

They also conduct experiments on a 2D toy dataset (25-Gaussians) and use feature vectors from ProjectedGAN for domain-agnostic evaluation.

Key Insights and Ideas

  • Diffusion as a Stable Noise Injector: The diffusion process naturally provides a smooth, controllable way to add noise, overcoming the practical difficulties of defining suitable noise distributions for instance noise injection in GANs.
  • Adaptive Difficulty for Discriminator: By adaptively adjusting the diffusion steps, the model can present a progressively harder task to the discriminator, preventing it from overfitting too early and ensuring it provides useful learning signals to the generator.
  • Non-Leaking Augmentation: The diffusion process is shown to be a "non-leaking" augmentation, meaning it doesn't introduce artifacts or biases that could harm generation quality, a problem sometimes seen with other augmentation techniques.
  • Theoretical Guarantees: The paper provides theoretical analysis showing that Diffusion-GAN's objective function is continuous and differentiable, ensuring that the generator always receives useful gradients.

Key Results

Diffusion-GAN consistently improves the performance of various GAN baselines:

  • Improved Fidelity and Diversity: Diffusion-GAN, when applied to models like StyleGAN2, ProjectedGAN, and InsGen, achieves state-of-the-art results on multiple benchmark datasets, outperforming baselines in both Fréchet Inception Distance (FID) for fidelity and Recall for diversity. For example, on CIFAR-10, Diffusion StyleGAN2 achieved an FID of 3.19 and Recall of 0.58, outperforming StyleGAN2 (FID 8.32, Recall 0.41).
  • Data Efficiency: The method significantly boosts performance in data-limited scenarios. For instance, Diffusion InsGen on FFHQ with only 5k images achieved an FID of 16.43, outperforming InsGen (18.21).
  • Domain Agnosticism: Applying Diffusion-GAN to feature vectors in ProjectedGAN also showed clear improvements in FID and Recall, demonstrating its applicability beyond image pixels.
  • Training Stability: Experiments show that Diffusion-GAN leads to more stable training, with adaptive diffusion strategies converging faster and reaching lower final FID scores.

Limitations and Future Work

  • Mixing Procedure Optimization: The paper notes that while the "priority mixing" strategy for sampling diffusion steps is intuitive, an ablation study suggests that "uniform mixing" might perform better on certain datasets. Optimizing this mixing procedure for different datasets is suggested as future work.
  • Computational Cost: While generally comparable to baselines, the diffusion process itself adds some computational overhead. The paper mentions that for StyleGAN2, the added memory cost is negative and training time cost is negative, while inference time is zero, suggesting efficiency gains in some aspects.
  • Further Investigation: The authors suggest that further investigation into optimizing the mixing procedure and exploring more satisfying possibilities for differentiable and invertible mapping functions could be beneficial.

Summary updated

Abstract

Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.
BibTeX
@inproceedings{wang2023diffusion,
  author = {Zhendong Wang and Huangjie Zheng and Pengcheng He and Weizhu Chen and Mingyuan Zhou},
  title = {{Diffusion-GAN: Training GANs with Diffusion}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=HZf7UbpWHuA},
}
Figure