Abstract
Generative adversarial networks (GANs), famous for the capability of learning complex underlying data distribution, are however known to be tricky in the training process, which would probably result in mode collapse or performance deterioration. Current approaches of dealing with GANs' issues almost utilize some practical training techniques for the purpose of regularization, which on the other hand undermines the convergence and theoretical soundness of GAN. In this paper, we propose to stabilize GAN training via a novel particle-based variational inference -- Langevin Stein variational gradient descent (LSVGD), which not only inherits the flexibility and efficiency of original SVGD but aims to address its instability issues by incorporating an extra disturbance into the update dynamics. We further demonstrate that by properly adjusting the noise variance, LSVGD simulates a Langevin process whose stationary distribution is exactly the target distribution. We also show that LSVGD dynamics has an implicit regularization which is able to enhance particles' spread-out and diversity. At last we present an efficient way of applying particle-based variational inference on a general GAN training procedure no matter what loss function is adopted. Experimental results on one synthetic dataset and three popular benchmark datasets -- Cifar-10, Tiny-ImageNet and CelebA validate that LSVGD can remarkably improve the performance and stability of various GAN models.
1. Introduction
Generative Adversarial Networks (GANs) are powerful tools for learning complex data distributions through a minimax game between a generator and a discriminator. However, they are notoriously difficult to train, often suffering from vanishing gradients and "mode collapse," where the generator produces a limited, repetitive subset of the target distribution.
Recent attempts to address these issues include Stein-GAN, which utilizes Stein Variational Gradient Descent (SVGD) to frame training as a Bayesian inference problem. While promising, SVGD introduces its own challenge: particle degeneracy. In high-dimensional spaces, the repulsive force intended to maintain diversity often fails, causing particles to collapse into low-variance clusters, effectively reintroducing mode collapse.
2. Langevin Stein Variational Gradient Descent (LSVGD)
To overcome particle degeneracy, we propose Langevin Stein Variational Gradient Descent (LSVGD). This method enhances the standard SVGD framework by injecting controlled Gaussian noise into the particle update dynamics.
2.1 Noise Injection and Langevin Dynamics
Unlike the deterministic updates of SVGD, LSVGD adds zero-centered Gaussian noise to each particle. We prove that by properly adjusting the noise variance, the particles follow a Langevin process, ensuring the stationary distribution matches the target distribution. This noise acts as a "kick" that prevents particles from becoming trapped in low-variance regions.
2.2 Implicit Regularization and GAN Integration
LSVGD introduces a mathematical regularization term that penalizes low-variance traps, forcing particles to explore the probability space more effectively. In a GAN setting, we treat the discriminator's feature extractor as a source of particles. The LSVGD update rule is applied to gradients at the outermost feature layer, and these refined signals are backpropagated through the network.
3. Experimental Results
We evaluated LSVGD across a synthetic bimodal distribution and three benchmark datasets: Cifar-10, Tiny-ImageNet, and CelebA.
3.1 Robustness and Diversity
A key finding is LSVGD's robustness to hyperparameters, specifically the kernel bandwidth ($\gamma$). While SVGD performance degrades significantly if $\gamma$ is not perfectly tuned, LSVGD maintains low KL-divergence and high variance across a wide range of values.
3.2 Benchmark Performance
- Inception Score (IS): On Cifar-10, LSVGD consistently outperformed SGD and SVGD. For instance, applying LSVGD to Self-Attention GAN (SAGAN) improved the IS from 7.80 to 8.87.
- Visual Quality: Qualitative results on CelebA show that LSVGD generates sharp, realistic faces with higher feature variance than competing methods, confirming its ability to avoid mode collapse.
4. Conclusion and Future Work
LSVGD provides a theoretically sound and practically robust method for stabilizing GAN training. By bridging particle-based variational inference with Langevin dynamics, it ensures greater sample diversity and easier hyperparameter tuning. Future work will address the computational overhead of particle interactions and explore more sophisticated noise variance estimation techniques to further scale the method to ultra-high-resolution image generation.
Summary updated