Improved BiGAN training with marginal likelihood equalization

Pablo Sánchez-Martín, Pablo M. Olmos, Fernando Pérez-CruzarXiv 2019 · arXiv

Introduction

Bidirectional Generative Adversarial Networks (BiGANs) extend the standard GAN framework by incorporating an encoder $E$ alongside the generator $G$ and discriminator $D$. This architecture enables the learning of inverse mappings from the data space $x$ to the latent space $z$, which is essential for representation learning. However, BiGANs frequently suffer from two primary failure modes: a mismatch between the encoded latent distribution and the prior, and a "simplicity bias" where the model over-represents easily modeled data at the expense of diversity. This paper introduces two regularization techniques—Prior Regularization and Marginal Likelihood Equalization (MLeq)—to address these challenges.

Addressing the Latent Space Mismatch

In high-dimensional spaces, a Gaussian prior distribution concentrates its mass on a thin spherical shell (the "typical set") rather than at the origin. Standard BiGAN training often fails to constrain the encoder to this region, resulting in real images being mapped to "untypical" latent codes that the generator rarely encounters during inference.

To rectify this, we propose Prior Regularization (P-MDGAN). By adding a penalty term that forces the norm of encoded vectors $\|E(x)\|_2$ toward the expected norm of the prior distribution ($\sqrt{D}$), we ensure that the encoder and generator operate within the same latent manifold. As shown in the comparison below, this regularization effectively aligns the distribution of encoded norms with the prior.

Distribution of latent norms for MDGAN (a) and P-MDGAN (b), demonstrating the alignment of encoded norms with the prior distribution through regularization.

Marginal Likelihood Equalization (MLeq)

GANs naturally gravitate toward modeling simpler data patterns, such as plain backgrounds, while struggling with complex textures or rare features. This leads to a lack of diversity in generated samples. We introduce Marginal Likelihood Equalization (MLeq) to combat this bias by moving from uniform to non-uniform mini-batch sampling.

The PSNR Proxy

Calculating the exact marginal likelihood for every image is computationally prohibitive. However, our empirical analysis reveals a strong correlation between an image's marginal likelihood and its reconstruction quality, measured via Peak Signal-to-Noise Ratio (PSNR). By using PSNR as a computationally efficient proxy, we can identify "hard-to-learn" images—those with low reconstruction quality—and sample them more frequently during training.

Scatter plot showing the high correlation between log-likelihood and PSNR, which justifies using PSNR as a proxy for identifying poorly modeled samples.

Results and Discussion

The combined approach, termed EP-MDGAN, was evaluated on CIFAR10, Fashion MNIST, and CelebA datasets. The results demonstrate significant improvements in both sample quality and diversity:

  • Quantitative Metrics: On CIFAR10, the Fréchet Inception Distance (FID) improved from 54.79 (baseline) to 37.57. On Fashion MNIST, the FID dropped from 18.85 to 8.83.
  • Diversity (Recall): The method achieved a more than two-fold increase in Recall, indicating that the model successfully captured a much wider variety of the original data distribution.
  • Visual Fidelity: In CelebA experiments, the model generated complex features like glasses and accessories that were entirely missing in baseline generations.

While the current non-uniform sampling strategy relies on a heuristic ranking, these results prove that equalizing the model's attention across the data distribution is a powerful tool for improving generative performance and mitigating mode collapse.

Summary updated

Abstract

We propose a novel training procedure for improving the performance of generative adversarial networks (GANs), especially to bidirectional GANs. First, we enforce that the empirical distribution of the inverse inference network matches the prior distribution, which favors the generator network reproducibility on the seen samples. Second, we have found that the marginal log-likelihood of the samples shows a severe overrepresentation of a certain type of samples. To address this issue, we propose to train the bidirectional GAN using a non-uniform sampling for the mini-batch selection, resulting in improved quality and variety in generated samples measured quantitatively and by visual inspection. We illustrate our new procedure with the well-known CIFAR10, Fashion MNIST and CelebA datasets.
BibTeX
@misc{sanchezmartin2019improved,
  author = {Pablo Sánchez-Martín and Pablo M. Olmos and Fernando Pérez-Cruz},
  title = {{Improved BiGAN training with marginal likelihood equalization}},
  year = {2019},
  eprint = {1911.01425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.01425},
}
Figure