Reparameterized Sampling for Generative Adversarial Networks

Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

TL;DR

This paper introduces REP-GAN, a novel sampling method that significantly improves the sample efficiency and quality of Generative Adversarial Networks (GANs). By reparameterizing the sampling process in the latent space, REP-GAN overcomes the limitations of previous methods, leading to more diverse and realistic generated samples.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) have shown remarkable success in generating realistic images. However, a persistent challenge is the quality and efficiency of the sampling process after the GAN has been trained. Existing methods often rely on independent proposals for sampling, which can lead to slow convergence (poor sample efficiency) and artifacts in the generated samples. This is because these methods struggle to effectively explore the complex, high-dimensional data distribution learned by the GAN. The paper aims to address this by developing a more efficient and effective sampling strategy that leverages the underlying structure of GANs.

Methods

The core innovation of REP-GAN is its reparameterized proposal mechanism. Instead of directly proposing new samples in the high-dimensional data space (X), REP-GAN proposes new samples in the lower-dimensional latent space (Z). This is achieved by:

  1. Structured Dependent Proposal: The method defines a dependent proposal in the latent space, q(z'|zk), where z' is the proposed latent variable and zk is the current latent variable. This is a departure from previous methods that used independent proposals.
  2. Latent-Level Transition: The proposed latent variable z' is then pushed forward through the generator G to obtain a sample proposal x' = G(z'). This effectively reparameterizes the sample-level transition x -> x' into a latent-level transition z -> z'.
  3. Tractable Metropolis-Hastings (MH) Criterion: A key theoretical contribution is showing that this reparameterized proposal allows for a tractable MH acceptance criterion. This criterion, CREP(x', xk), successfully cancels out unknown data density terms, making the sampling process efficient and effective.
  4. Incorporation of Existing Mechanisms: REP-GAN unifies and extends existing sampling mechanisms by incorporating Markov Chain Monte Carlo (MCMC), acceptance-rejection steps, and latent gradient-based proposals.

The paper also proposes a Latent Langevin Monte Carlo (L2MC) proposal as a practical implementation of REP-GAN, which leverages the gradient information in the latent space for more efficient exploration.

Datasets

The paper evaluates REP-GAN on the following datasets:

  • Synthetic Datasets:
    • Swiss Roll dataset (used for visualizing manifold learning)
    • 25-Gaussians dataset (used for evaluating multi-modal distribution modeling)
  • Real-world Image Datasets:
    • CIFAR-10
    • CelebA

The experiments use pre-trained GANs (DCGAN and WGAN) on these datasets.

Key Insights and Ideas

  • Latent Space is Key: The paper highlights that operating in the lower-dimensional latent space of a GAN is significantly easier and more effective for sampling than working directly in the high-dimensional data space. This is because the latent space often has a "better landscape" for proposals.
  • Dependent Proposals Improve Efficiency: By using dependent proposals in the latent space, REP-GAN achieves higher acceptance rates compared to independent proposals, leading to faster mixing of the Markov chains and better sample efficiency.
  • Tractable MH Criterion is Crucial: The ability to derive a tractable MH acceptance criterion for the reparameterized proposal is a major breakthrough, enabling the practical application of dependent proposals in GAN sampling.
  • Unification of Sampling Techniques: REP-GAN provides a unified framework that can encompass and improve upon existing sampling methods for GANs.
  • MH Correction Prevents Artifacts: The MH rejection step in REP-GAN plays a vital role in preventing the misbehavior and artifacts that can arise from large step sizes or numerical errors in gradient-based sampling methods.

Key Results

  • Improved Sample Quality: REP-GAN consistently achieves better sample quality, as measured by Inception Scores and Fréchet Inception Distances (FID), compared to state-of-the-art methods like MH-GAN and DDLS on CIFAR-10 and CelebA datasets. For example, on CIFAR-10 with DCGAN, REP-GAN achieved an Inception Score of 3.541, outperforming MH-GAN (3.225) and DDLS (3.152).
  • Higher Sample Efficiency: The paper demonstrates significantly higher acceptance ratios for REP-GAN compared to MH-GAN. For instance, on CIFAR-10, MH-GAN's acceptance ratio was often below 0.05, while REP-GAN achieved ratios between 0.2 and 0.5. This indicates faster convergence and more efficient exploration of the target distribution.
  • Robustness to Step Size: REP-GAN is more robust to larger step sizes in Langevin dynamics compared to methods like DDLS, thanks to its MH correction step, which prevents divergence and artifacts.
  • Preserves Diversity: On multi-modal datasets, REP-GAN is shown to preserve sample diversity better than DDLS, which tends to concentrate too much on mode centers.

Limitations and Future Work

  • Assumption of Perfect Discriminator: The theoretical analysis relies on the assumption that the discriminator provides a perfect estimate of the density ratio. The authors acknowledge that this is a strong assumption and unlikely to hold exactly in practice, as the generator and discriminator are trained adversarially. However, they argue that this assumption still captures significant information about the density ratio, leading to improvements.
  • Future Directions: The authors suggest several avenues for future work:
  • Developing mechanisms for more accurate density ratio estimation.
  • Relaxing the assumptions made in the analysis.
  • Establishing estimation error bounds for the density ratio.
  • Exploring GANs as a platform for developing new sampling methods.

Summary updated

Abstract

Recently, sampling methods have been successfully applied to enhance the sample quality of Generative Adversarial Networks (GANs). However, in practice, they typically have poor sample efficiency because of the independent proposal sampling from the generator. In this work, we propose REP-GAN, a novel sampling method that allows general dependent proposals by REParameterizing the Markov chains into the latent space of the generator. Theoretically, we show that our reparameterized proposal admits a closed-form Metropolis-Hastings acceptance ratio. Empirically, extensive experiments on synthetic and real datasets demonstrate that our REP-GAN largely improves the sample efficiency and obtains better sample quality simultaneously.
BibTeX
@inproceedings{wang2021reparameterized,
  author = {Yifei Wang and Yisen Wang and Jiansheng Yang and Zhouchen Lin},
  title = {{Reparameterized Sampling for Generative Adversarial Networks}},
  booktitle = {ECML/PKDD},
  year = {2021},
  doi = {10.1007/978-3-030-86523-8\_30},
}
Figure