Abstract
Generative Adversarial Networks (GANs) have become a widely popular framework for generative modelling of high-dimensional datasets. However, their training is well-known to be difficult. This work provides a rigorous mathematical explanation for why GANs suffer from vanishing gradients, pointing to the inevitable memorization that occurs when using finite, empirical training data. To solve this, the authors propose Kernel GANs, which smooth the Jensen-Shannon Divergence (JSD) using Kernel Density Estimation (KDE), effectively replacing the neural network discriminator with a closed-form, non-parametric mathematical objective.
Introduction: The Empirical Trap
Generative Adversarial Networks (GANs) are notorious for being difficult to train, frequently suffering from mode collapse and vanishing gradients. Previous theoretical work suggested that if a discriminator becomes "too good," it perfectly separates real and fake data, leaving no useful gradient for the generator to learn from.
This paper argues that a major source of confusion in GAN literature is the failure to distinguish between the theoretical data distribution (continuous and infinite) and the empirical data distribution (the finite dataset used in practice). The authors prove that when training on finite samples, an optimal discriminator will simply place sharp spikes (Dirac functions) exactly on the training examples. Unless a generated sample randomly lands on the exact coordinates of a training image, the gradient will vanish. This research seeks a principled way to smooth out this objective to keep gradients flowing.
Methodology: Kernel GANs
To prevent the discriminator from perfectly memorizing finite data points, the authors propose adding noise to the discriminator's inputs and utilizing Kernel Density Estimation (KDE).
Closed-Form Discriminator
By smoothing the distributions, the authors prove that the optimal discriminator can be written as a ratio of KDEs. Because the optimal discriminator is now a known mathematical formula, the authors completely discard the discriminator neural network. Instead, they plug this KDE formula directly into the generator's loss function. This is akin to "unrolling" the discriminator to infinite steps until it reaches closed-form optimality.
Feature Space Mapping
Standard kernels (like Radial Basis Functions) struggle to measure distances in high-dimensional pixel space. To scale their method to complex images, the authors first pass the data through a convolutional autoencoder. The Kernel GAN objective is then computed in this lower-dimensional latent space.
Experiments and Results
The authors evaluate their approach on standard generative modeling benchmarks, including Mixture-of-Gaussian (MOG), MNIST, CIFAR-10, and CelebA.
Stable Training on Toy Data
The Kernel GAN successfully avoids vanishing gradients. This is demonstrated visually on the MOG toy dataset, where the generator smoothly collapses onto the true data modes as the kernel bandwidth is annealed.
Image Generation Performance
On the MNIST dataset, the Deep Convolutional (DC) Kernel GAN achieved a strong balance of sample fidelity and diversity.
For CIFAR-10, the model achieved an Inception Score of 4.22 (± 0.02). While this significantly outperformed standard Generative Moment Matching Networks (GMMNs) at 3.94, it fell slightly short of state-of-the-art kernel-based models like MMD-GAN (6.17) and WGAN (5.88).
Discussion
Key Insights
- The Vanishing Gradient Problem: This is an inevitable consequence of using finite training data. An optimal empirical discriminator will always perfectly separate the data, killing the gradient.
- Bypassing the Min-Max Game: By using non-parametric KDEs, we don't need to train a discriminator network via gradient descent. We can optimize the generator directly against an estimate of the true Jensen-Shannon Divergence.
- Bandwidth Control: The kernel bandwidth ($\sigma$) acts as a crucial control mechanism. Starting with a large bandwidth provides broad, smooth gradients, while gradually annealing it forces the generator to learn fine-grained details.
Limitations and Future Work
The core Kernel GAN formulation struggles with raw, high-dimensional color images, relying on a pre-learned autoencoder. This complicates the theoretical purity of the approach. Future work should focus on designing better kernels, developing automated strategies for annealing the kernel bandwidth, and investigating the interaction between the autoencoder's feature mapping and the mathematical guarantees.
Summary updated