FIS-GAN: GAN with Flow-based Importance Sampling

Shiyu Yi, Donglin Zhan, Zhengyang Geng, Wenqing Zhang, Chang XuarXiv 2019 · OpenReview

Abstract

Generative Adversarial Networks (GAN) training process, in most cases, apply Uniform or Gaussian sampling methods in the latent space, which probably spends most of the computation on examples that can be properly handled and easy to generate. Theoretically, importance sampling speeds up stochastic optimization in supervised learning by prioritizing training examples. In this paper, we explore the possibility of adapting importance sampling into adversarial learning. We use importance sampling to replace Uniform and Gaussian sampling methods in the latent space and employ normalizing flow to approximate latent space posterior distribution by density estimation. Empirically, results on MNIST and Fashion-MNIST demonstrate that our method significantly accelerates GAN's optimization while retaining visual fidelity in generated samples.

Introduction

Training Generative Adversarial Networks (GANs) is notoriously difficult and computationally expensive. Traditionally, generators draw random noise from simple distributions, such as standard Normal or Uniform distributions. However, this approach treats all regions of the latent space equally, often leading the model to waste computation on "easy" examples it has already mastered.

In supervised learning, importance sampling is a well-established technique to prioritize challenging training examples and accelerate optimization. FIS-GAN adapts this concept to the unsupervised, adversarial setting. The core challenge lies in identifying "important" regions of the latent space and efficiently sampling from them.

Methods: Flow-based Importance Sampling

FIS-GAN introduces a dynamic feedback loop to refine the generator's focus within the latent space. The process involves several key components:

  1. Importance Metric via Jacobian Norm: The "importance" of a latent point is quantified by the Jacobian norm of the generator. A large norm indicates high sensitivity—small changes in the latent input cause large changes in the output. These regions are considered "harder" for the model to map accurately.
  2. Density Estimation with Normalizing Flows: Because the resulting importance density is complex and high-dimensional, FIS-GAN employs Normalizing Flows (e.g., Real-NVP, MAF, or IAF). The flow learns to transform a simple base distribution into the complex "importance" distribution derived from the Jacobian norms.
  3. Dynamic Re-sampling: The generator draws noise through the trained flow for subsequent iterations, forcing it to focus on "hard" instances. This importance map is periodically refreshed (e.g., every 10 to 50 steps) to evolve alongside the generator's improvements.
Overview of the FIS-GAN architecture, illustrating the feedback loop from Jacobian computation to flow-based importance sampling.

The authors found that using the Frobenius norm for Jacobian calculations provides an optimal balance between performance and computational complexity compared to more expensive alternatives like the Nuclear norm.

Experimental Results

The researchers evaluated FIS-GAN on the MNIST and Fashion-MNIST benchmarks using the Fréchet Inception Distance (FID) as the primary metric for image quality and diversity.

Quantitative Performance

FIS-GAN demonstrates a significant acceleration in convergence. On Fashion-MNIST, FIS-GAN achieved an FID of 4.36 in just 500 steps, whereas the baseline DCGAN remained at 10.52. Similar gains were observed on MNIST, where FIS-GAN reached an FID of 197.02 at 500 steps, outperforming the vanilla GAN's 246.35.

Comparison of Fréchet Inception Distance (FID) over training steps for vanilla GAN vs. FIS-GAN on MNIST.

Qualitative Analysis

Visual inspection confirms that FIS-GAN enters a stable training interval much faster. Generated samples for both digits and clothing items appear recognizable and smooth much earlier in the training process than those from baseline models. While various flow architectures were tested, MAF (Masked Autoregressive Flow) yielded the most significant improvements, though Real-NVP is noted for its practical balance of speed and performance.

Discussion and Future Work

While FIS-GAN reduces the total number of training steps required for convergence, each "importance update" step introduces additional computational overhead due to Jacobian calculations and flow training. Future research is needed to determine how this approach scales to high-resolution datasets (e.g., ImageNet), where these calculations become more taxing. Additionally, the sensitivity to hyperparameters like the update frequency ($t$) suggests that automated tuning could further enhance the robustness of the method.

Summary updated

Abstract

Generative Adversarial Networks (GAN) training process, in most cases, apply Uniform or Gaussian sampling methods in the latent space, which probably spends most of the computation on examples that can be properly handled and easy to generate. Theoretically, importance sampling speeds up stochastic optimization in supervised learning by prioritizing training examples. In this paper, we explore the possibility of adapting importance sampling into adversarial learning. We use importance sampling to replace Uniform and Gaussian sampling methods in the latent space and employ normalizing flow to approximate latent space posterior distribution by density estimation. Empirically, results on MNIST and Fashion-MNIST demonstrate that our method significantly accelerates GAN's optimization while retaining visual fidelity in generated samples.
BibTeX
@misc{yi2019fis,
  author = {Shiyu Yi and Donglin Zhan and Zhengyang Geng and Wenqing Zhang and Chang Xu},
  title = {{FIS-GAN: GAN with Flow-based Importance Sampling}},
  year = {2019},
  eprint = {1910.02519},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.02519},
}
Figure