Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, Yoshua Bengio

Introduction

Generative Adversarial Networks (GANs) have become a cornerstone of high-resolution image synthesis. Traditionally, these models are utilized by passing a noise vector through a trained generator. However, this standard approach often discards the discriminator after training, despite the discriminator containing critical information about the generator's shortcomings. This paper posits that a GAN implicitly defines an Energy-Based Model (EBM) and introduces Discriminator Driven Latent Sampling (DDLS)—a technique that leverages the discriminator's gradients to refine samples in the latent space, significantly enhancing both image quality and diversity.

The Implicit EBM Framework

The fundamental insight of this work is that the generator and discriminator together define a "secret" EBM. The authors demonstrate that the true data distribution $p_{data}(x)$ can be approximated by a distribution $p(x)$ that combines the generator's implicit density $p_G(x)$ with the discriminator's learned scores. Specifically, under an optimal discriminator, the distribution $p(x) \propto p_G(x) \exp(D(x))$ (where $D(x)$ is the discriminator's logit) perfectly matches the target data distribution. This realization transforms the discriminator from a mere training adversary into a powerful post-processing tool for density ratio estimation.

Discriminator Driven Latent Sampling (DDLS)

While sampling from EBMs typically requires computationally expensive Markov Chain Monte Carlo (MCMC) in high-dimensional pixel space, the authors propose performing this refinement in the low-dimensional latent space.

The Algorithm

Instead of a single forward pass, DDLS employs Langevin dynamics on the latent vector $z$. By using the gradients of the discriminator's output with respect to the latent vector, the algorithm iteratively "pushes" $z$ toward regions that the discriminator identifies as more realistic. To address the "zero support" problem—where the generator fails to cover certain modes—the authors introduce an extended latent space by injecting small amounts of Gaussian noise into the generator's output, allowing the MCMC process to explore slightly beyond the learned manifold.

Empirical Evaluation

Synthetic Data and Mode Recovery

The effectiveness of DDLS was first tested on synthetic datasets, including a 25-Gaussian mixture and the "Swiss Roll." In these controlled environments, vanilla generators often suffer from mode collapse or produce samples between modes (artifacts).

Comparison of sampling methods on a 2D Gaussian mixture. DDLS (labeled DLS) significantly improves mode recovery and reduces artifacts compared to the vanilla generator and the DOT baseline.

As shown in Figure 4, DDLS (DLS) successfully recovers 98% of high-quality modes, a substantial improvement over the 70% achieved by the vanilla generator. The refinement process effectively cleans up the distribution, pulling samples toward the true centers of the Gaussian modes.

DDLS refinement on the Swiss Roll dataset, demonstrating the algorithm's ability to accurately follow the underlying data manifold.

Real-World Image Synthesis

The method was further validated on standard benchmarks including CIFAR-10, CelebA, and ImageNet. On CIFAR-10, applying DDLS to a pre-trained SN-GAN improved the Inception Score (IS) from 8.22 to 9.09, rivaling the performance of more complex class-conditional models. Similar gains were observed on ImageNet, where the IS rose from 36.8 to 40.2 without any additional training of the base models.

Discussion and Limitations

DDLS provides a "free" performance boost for existing GANs by treating the discriminator as a correction signal. By operating in latent space, it achieves faster mixing and better scalability than previous pixel-space methods. However, the technique introduces computational overhead, as each generated image requires multiple forward and backward passes through the discriminator. Future work may explore integrating these EBM insights into other architectures like VAEs or optimizing the MCMC iterations for real-time applications.

Summary updated

BibTeX
@inproceedings{che2020your,
  author = {Tong Che and Ruixiang Zhang and Jascha Sohl-Dickstein and Hugo Larochelle and Liam Paull and Yuan Cao and Yoshua Bengio},
  title = {{Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=G0UOPnVlIri},
}
Figure