Co-Generation with GANs using AIS based HMC

Tiantian Fang, Alexander G. Schwing

Abstract

Co-generation involves inferring missing data components using pre-trained Generative Adversarial Networks (GANs). While GANs excel at generation, inverting their latent space for tasks like inpainting is challenging due to the highly non-convex and "ragged" nature of the energy landscape in well-trained models. We present a method combining Annealed Importance Sampling (AIS) and Hamiltonian Monte Carlo (HMC) to robustly navigate these complex landscapes. Our approach demonstrates superior performance in finding optimal latent representations compared to standard gradient descent, particularly as GAN training progresses and the inversion task becomes more difficult.

Introduction

Generative Adversarial Networks (GANs) have revolutionized data generation. However, "co-generation"—the task of filling in missing parts of an image, such as inpainting or super-resolution—requires finding a latent vector $z$ that matches observed pixels. The standard approach uses gradient-based optimization to search the latent space. Unfortunately, as GANs improve, their latent spaces become increasingly non-convex and "ragged," causing gradient descent to frequently trap in local optima. This paper addresses these challenges by replacing simple optimization with a sophisticated sampling-based approach.

Methodology: AIS-based HMC

To navigate the complex energy landscape of a GAN's latent space, we utilize a combination of two powerful statistical tools:

  1. Hamiltonian Monte Carlo (HMC): By introducing auxiliary momentum and simulating physical dynamics, HMC can traverse energy barriers that would stop standard gradient descent. It effectively "rolls over" the hills of the non-convex loss landscape.
  2. Annealed Importance Sampling (AIS): AIS manages complexity by starting with a simple Gaussian prior and gradually annealing towards the complex posterior distribution. This step-by-step transition guides the search toward the global optimum, making the process robust to initialization.
Evolution of the latent space landscape and inversion difficulty as GAN training progresses. As the model improves (higher iterations), the landscape becomes more "ragged," making standard gradient descent techniques fail to reach the global minimum.

Experimental Setup

The method was evaluated using Progressive GANs across several domains:

  • Synthetic 2D Data: A 5-Gaussian mixture to visualize energy barrier crossing.
  • CelebA & LSUN Bedrooms: Image inpainting tasks at various resolutions.
  • CelebA-HQ: Single Image Super-Resolution (SISR) upscaling to $1024\times1024$.

Key Insights

A critical finding is the inverse relationship between GAN quality and inversion ease. As a GAN becomes better at modeling data, its latent space becomes more "ragged," significantly increasing the likelihood of gradient descent failure. Furthermore, while standard inversion is highly sensitive to initialization, the AIS-HMC approach is robust, typically requiring only a single initialization due to its guided annealing process.

Results

  • Robustness: In synthetic tests, AIS-HMC achieved a 100% success rate in finding optimal modes, whereas gradient descent consistently failed.
  • Inpainting Performance: On CelebA and LSUN datasets, the proposed method outperformed gradient descent baselines in both MSE and MSSIM metrics.
  • Efficiency: On LSUN, AIS-HMC with 5 initializations surpassed gradient descent using 5,000 initializations.
  • Super-Resolution: The method successfully generated high-frequency details for $1024\times1024$ images, maintaining visual coherence with low-resolution inputs.

Conclusion

While computationally more intensive than simple gradient descent due to the iterative nature of MCMC sampling, AIS-based HMC provides a robust solution for GAN inversion in complex landscapes. Future work may explore accelerating the sampling process or hybridizing this approach with learned encoder-based methods.

Summary updated

BibTeX
@inproceedings{fang2019co,
  author = {Tiantian Fang and Alexander G. Schwing},
  title = {{Co-Generation with GANs using AIS based HMC}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure