Introduction
Generative Adversarial Networks (GANs) have become a cornerstone of high-resolution image synthesis. Traditionally, these models are utilized by passing a noise vector through a trained generator. However, this standard approach often discards the discriminator after training, despite the discriminator containing critical information about the generator's shortcomings. This paper posits that a GAN implicitly defines an Energy-Based Model (EBM) and introduces Discriminator Driven Latent Sampling (DDLS)—a technique that leverages the discriminator's gradients to refine samples in the latent space, significantly enhancing both image quality and diversity.
The Implicit EBM Framework
The fundamental insight of this work is that the generator and discriminator together define a "secret" EBM. The authors demonstrate that the true data distribution $p_{data}(x)$ can be approximated by a distribution $p(x)$ that combines the generator's implicit density $p_G(x)$ with the discriminator's learned scores. Specifically, under an optimal discriminator, the distribution $p(x) \propto p_G(x) \exp(D(x))$ (where $D(x)$ is the discriminator's logit) perfectly matches the target data distribution. This realization transforms the discriminator from a mere training adversary into a powerful post-processing tool for density ratio estimation.
Discriminator Driven Latent Sampling (DDLS)
While sampling from EBMs typically requires computationally expensive Markov Chain Monte Carlo (MCMC) in high-dimensional pixel space, the authors propose performing this refinement in the low-dimensional latent space.
The Algorithm
Instead of a single forward pass, DDLS employs Langevin dynamics on the latent vector $z$. By using the gradients of the discriminator's output with respect to the latent vector, the algorithm iteratively "pushes" $z$ toward regions that the discriminator identifies as more realistic. To address the "zero support" problem—where the generator fails to cover certain modes—the authors introduce an extended latent space by injecting small amounts of Gaussian noise into the generator's output, allowing the MCMC process to explore slightly beyond the learned manifold.
Empirical Evaluation
Synthetic Data and Mode Recovery
The effectiveness of DDLS was first tested on synthetic datasets, including a 25-Gaussian mixture and the "Swiss Roll." In these controlled environments, vanilla generators often suffer from mode collapse or produce samples between modes (artifacts).
As shown in Figure 4, DDLS (DLS) successfully recovers 98% of high-quality modes, a substantial improvement over the 70% achieved by the vanilla generator. The refinement process effectively cleans up the distribution, pulling samples toward the true centers of the Gaussian modes.
Real-World Image Synthesis
The method was further validated on standard benchmarks including CIFAR-10, CelebA, and ImageNet. On CIFAR-10, applying DDLS to a pre-trained SN-GAN improved the Inception Score (IS) from 8.22 to 9.09, rivaling the performance of more complex class-conditional models. Similar gains were observed on ImageNet, where the IS rose from 36.8 to 40.2 without any additional training of the base models.
Discussion and Limitations
DDLS provides a "free" performance boost for existing GANs by treating the discriminator as a correction signal. By operating in latent space, it achieves faster mixing and better scalability than previous pixel-space methods. However, the technique introduces computational overhead, as each generated image requires multiple forward and backward passes through the discriminator. Future work may explore integrating these EBM insights into other architectures like VAEs or optimizing the MCMC iterations for real-time applications.
Summary updated