Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent

Dong Wang, Xiaoqian Qin, Fengyi Song, Li Cheng

TL;DR

Training Generative Adversarial Networks (GANs) is notoriously unstable, often leading to "mode collapse" where the generator produces limited, repetitive varieties of data. This paper introduces Langevin Stein Variational Gradient Descent (LSVGD), a novel particle-based inference method that injects controlled noise into the training process to prevent particles from clumping together, ensuring more diverse and stable image generation across various GAN architectures.

What Problem Does This Paper Address?

GANs operate as a minimax game between a generator and a discriminator. While powerful, they suffer from two major headaches: vanishing gradients (where the generator stops learning) and mode collapse (where the generator finds a "safe" output that fools the discriminator but lacks diversity).

Previous attempts to fix this include training "tricks" or complex regularization (like WGAN’s weight clipping), but these often sacrifice the theoretical convergence of the model. A newer approach, Stein-GAN, used Stein Variational Gradient Descent (SVGD) to treat training as a Bayesian inference problem. However, SVGD has its own flaw: particle degeneracy. In high-dimensional spaces, the "repulsive force" meant to keep samples diverse often fails, causing samples to collapse into low-variance clusters—essentially bringing mode collapse back through the side door.

Methods

The authors propose Langevin Stein Variational Gradient Descent (LSVGD). It builds on the standard SVGD framework but adds a crucial "disturbance" or noise term to the update dynamics.

  1. Noise Injection: Unlike standard SVGD, which uses a deterministic update, LSVGD injects zero-centered Gaussian noise into each particle.
  2. Langevin Dynamics: The authors prove that by properly adjusting the variance of this noise, the particles follow a Langevin process. This ensures that the stationary distribution of these particles is exactly the target distribution.
  3. Implicit Regularization: Mathematically, LSVGD introduces a regularization term that penalizes "low-variance traps." This forces particles to spread out and explore the probability space more effectively than standard SVGD.
  4. GAN Integration: To use this in a GAN, the authors treat the discriminator's feature extractor as a source of "particles." Instead of standard backpropagation (SGD), they apply the LSVGD update rule to the gradients at the outermost feature layer, then propagate those refined signals back through the network.

Datasets

The researchers evaluated LSVGD on four distinct setups:

  • Synthetic Dataset: A bimodal Gaussian distribution used to visualize particle behavior and sensitivity to parameters.
  • Cifar-10: 60,000 32x32 images across 10 categories (airplane, automobile, etc.).
  • Tiny-ImageNet: A subset of ImageNet with 200 classes and 100,000 training images.
  • CelebA: A large-scale face attributes dataset with over 200,000 celebrity images.

Key Insights and Ideas

  • Escaping the Trap: The most significant insight is the identification of "low-variance traps." Standard SVGD is highly sensitive to the "bandwidth" of its kernel; if the bandwidth is too small, particles stop repelling each other. LSVGD’s added noise acts as a "kick" that pushes particles out of these stagnant areas.
  • Theoretical Bridge: The paper successfully bridges the gap between particle-based variational inference and Stochastic Gradient MCMC (SGMCMC). By treating GAN training as a Bayesian inference task on a "moving target" (the evolving discriminator), they provide a more rigorous foundation for GAN stability.
  • Robustness to Hyperparameters: One of the biggest practical "aha" moments is that LSVGD is far less sensitive to the kernel bandwidth ($\gamma$) than SVGD. This makes it much easier to tune for different datasets.

Key Results

The authors tested LSVGD against standard SGD and SVGD across seven different GAN variants (including DCGAN, WGAN-GP, and SAGAN).

  • Inception Score (IS): On Cifar-10, LSVGD consistently outperformed both SGD and SVGD. For example, applying LSVGD to SAGAN raised the IS from 7.80 to 8.87.
  • Diversity (Variance): On the CelebA dataset, LSVGD produced a higher "feature variance" than other methods, confirming that it successfully generates a wider variety of faces and avoids mode collapse.
  • Visual Quality: Qualitative results (images) show that while SVGD often produces blurred images when parameters are sub-optimal, LSVGD maintains sharp, realistic details while ensuring sample diversity.

Limitations and Future Work

  • Computational Overhead: Particle-based methods like LSVGD are generally more computationally expensive than standard SGD because they require calculating interactions between multiple particles (samples) in a batch.
  • Noise Estimation: The method requires an empirical estimation of the noise variance ($V(x)$). The authors used a simple thresholding technique, but more sophisticated ways to estimate this variance could potentially improve performance further.
  • Scaling to Ultra-High Resolution: While tested on 64x64 images (Tiny-ImageNet/CelebA), the paper does not explore how the particle-based update scales to very high-resolution GANs (like StyleGAN), where the dimensionality of the feature space is significantly larger.

Summary updated

Abstract

Generative adversarial networks (GANs), which are famous for the capability of learning complex underlying data distribution, are, however, known to be tricky in the training process, which would probably result in mode collapse or performance deterioration. Current approaches of dealing with GANs’ issues almost utilize some practical training techniques for the purpose of regularization, which, on the other hand, undermines the convergence and theoretical soundness of GAN. In this article, we propose to stabilize GAN training via a novel particle-based variational inference—Langevin Stein variational gradient descent (LSVGD), which not only inherits the flexibility and efficiency of original SVGD but also aims to address its instability issues by incorporating an extra disturbance into the update dynamics. We further demonstrate that, by properly adjusting the noise variance, LSVGD simulates a Langevin process whose stationary distribution is exactly the target distribution. We also show that LSVGD dynamics has an implicit regularization, which is able to enhance particles’ spread-out and diversity. Finally, we present an efficient way of applying particle-based variational inference on a general GAN training procedure no matter what loss function is adopted. Experimental results on one synthetic data set and three popular benchmark data sets—Cifar-10, Tiny-ImageNet, and CelebA—validate that LSVGD can remarkably improve the performance and stability of various GAN models.
BibTeX
@article{wang2022stabilizing,
  author = {Dong Wang and Xiaoqian Qin and Fengyi Song and Li Cheng},
  title = {{Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=I3Pjp3gjJj},
}

Loading related content...

Figure