Introduction
This paper introduces a robust method for "co-generation"—the task of filling in missing parts of a data sample (such as image inpainting) using a pre-trained Generative Adversarial Network (GAN). By combining Annealed Importance Sampling (AIS) with Hamiltonian Monte Carlo (HMC), the authors successfully navigate the complex, "ragged" energy landscapes of well-trained GANs where standard gradient-based optimization typically fails.
The Challenge of Co-Generation
Generative models like GANs are excellent at creating high-quality data from scratch by mapping a simple latent vector ($z$) to a complex output ($x$). However, they are notoriously difficult to use for conditional generation or co-generation. Given a partial observation (e.g., an image with a missing patch), the goal is to find the specific latent vector $z$ that perfectly matches the observed part to seamlessly fill in the missing region.
The standard approach uses gradient descent (GD) to minimize the difference between the observed pixels and the generator's output. However, the authors point out a fundamental flaw: as a GAN becomes better trained, the mapping from latent space to image space becomes increasingly "ragged" and non-convex.
This creates a landscape full of local optima and "energy barriers" that trap standard gradient descent trajectories, leading to poor reconstructions and mismatched artifacts.
Proposed Method: AIS-HMC
To overcome the limitations of gradient descent, the authors propose an algorithm that shifts the perspective from finding the single best $z$ (optimization) to sampling from the posterior distribution of $z$ (sampling). Their approach relies on two key components:
- Annealed Importance Sampling (AIS): Instead of jumping straight into the complex, ragged distribution of the trained GAN, the algorithm starts with a simple, smooth distribution (the Gaussian prior). It then gradually "anneals" or transitions toward the target posterior distribution through a series of intermediate steps. This prevents the model from getting stuck in the first local optimum it encounters.
- Hamiltonian Monte Carlo (HMC): Within each annealing step, the algorithm uses HMC to sample the latent space. Unlike standard random-walk sampling, HMC uses the gradient of the distribution to simulate "physical momentum." This allows the sampler to traverse large distances in the latent space and "coast" over small energy barriers that would otherwise halt a gradient descent optimizer.
Unlike encoder-decoder models that must be trained for a specific task (like a fixed-size mask), this co-generation framework is highly flexible. It works on any arbitrary subset of the data at test time without requiring the generator to be retrained.
Key Insights and The "Success Paradox"
Interestingly, the paper highlights a "Success Paradox": the better a GAN is at generating realistic images, the harder it becomes to perform co-generation using gradient descent. The "folds" in the latent space become sharper as the model captures finer details, making the optimization landscape much more treacherous.
Because sampling is inherently more robust to these complex topologies than point optimization, the AIS-HMC approach is remarkably resilient. While gradient descent is highly sensitive to initialization in the latent space, AIS-HMC often requires only a single initialization to find a high-quality solution.
Results and Limitations
The authors evaluated their approach on several benchmarks, including synthetic 2D data, CelebA ($128 \times 128$ faces), LSUN ($256 \times 256$ bedrooms), and CelebA-HQ ($1024 \times 1024$ faces).
- Quantitative and Visual Superiority: On CelebA and LSUN, the proposed method significantly outperformed gradient descent baselines across Mean Squared Error (MSE) and Mean Structural Similarity (MSSIM) metrics. Qualitatively, AIS-HMC produces inpainting results that are structurally consistent with the ground truth, avoiding the blurry artifacts common with GD.
- Super-Resolution: The method successfully performed $8\times$ super-resolution ($128 \to 1024$ pixels) by treating the low-resolution image as the "observed" part of a downsampling distribution.
Limitations: The primary drawback of this approach is computational cost. Running hundreds of HMC loops with leapfrog integration is significantly slower than a few iterations of gradient descent. Additionally, performance is sensitive to hyperparameters like the annealing schedule and HMC step size. Finally, while effective for standard GAN latent spaces (e.g., 512 dimensions), scaling HMC sampling efficiency to extremely high-dimensional latent spaces remains an open challenge for future work.
Summary updated