Pairwise Augmented GANs with Adversarial Reconstruction Loss

Aibek Alanov, Max Kochurov, Daniil Yashkov, Dmitry P. VetrovarXiv 2018 · arXiv

Introduction

Generative modeling has traditionally been divided between two dominant paradigms: Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). While VAEs excel at learning structured latent representations through an encoder-decoder framework, they often produce blurry samples due to their reliance on pixel-wise reconstruction losses like $L_2$ distance. Conversely, GANs generate high-fidelity, sharp images but lack an inherent mechanism to map data back into the latent space.

Hybrid models attempt to unify these strengths but frequently struggle with the "pixel-perfect" trap. Standard reconstruction losses penalize any spatial deviation, forcing the model to expend capacity on irrelevant low-level details rather than semantic content. This paper introduces Pairwise Augmented GANs (PAGANs), which utilize an adversarial reconstruction loss to focus on high-level features rather than exact pixel matches.

Methodology: Pairwise Augmentation

The core innovation of PAGAN is the introduction of an Augmenter function and a pairwise discriminator. Instead of a simple point-wise comparison between an image $x$ and its reconstruction $G(E(x))$, the model compares pairs of images.

The architectural evolution from standard GANs and BiGANs to the proposed PAGAN framework.

As shown in the architecture diagram, the PAGAN framework employs a discriminator $D(x, y)$ that evaluates pairs. A "Real Pair" consists of an original image and its augmented version (e.g., a small crop or shift), while a "Fake Pair" consists of the original image and its reconstruction. By training the discriminator to accept augmented versions as "real," the generator is incentivized to produce reconstructions that preserve semantic content even if they are spatially shifted. This stochasticity prevents the discriminator from becoming "too perfect," thereby stabilizing the adversarial training process.

Experimental Setup and Metrics

The model was benchmarked on MNIST, CIFAR10, and CelebA datasets. To quantify the quality of semantic reconstruction, the authors introduced Reconstruction Inception Dissimilarity (RID). Unlike traditional metrics like Root Mean Square Error (RMSE), which penalize any pixel-level change, RID uses a pre-trained Inception network to compare feature distributions. This ensures that a reconstruction is judged on its content rather than its exact alignment.

Key Results

PAGAN demonstrates significant improvements in both sample quality and reconstruction fidelity. On the CIFAR10 dataset, it achieved a state-of-the-art Fréchet Inception Distance (FID) of 32.84, surpassing other hybrid models such as AGE, ALI, and WAE-GAN.

Visual comparison of samples and reconstructions between PAGAN and the AGE baseline on CIFAR10.

Quantitatively, PAGAN's RID score of 13.01 significantly outperformed the AGE model's 19.02. Visual inspection confirms that PAGAN reconstructions are sharper and semantically more consistent with the input images. Ablation studies further revealed that both the adversarial loss and the augmentation step are critical; removing either component leads to a substantial drop in performance.

Conclusion and Future Work

PAGAN successfully bridges the gap between GANs and VAEs by prioritizing semantic invariance over pixel-wise accuracy. While the current implementation relies on manual augmentation choices like shifts and crops, future research could explore learned augmentations or more sophisticated architectures like ResNets to further enhance performance. Additionally, investigating the sensitivity of the model to different types of noise and transformations remains an open area for exploration.

Summary updated

Abstract

We propose a novel autoencoding model called Pairwise Augmented GANs. We train a generator and an encoder jointly and in an adversarial manner. The generator network learns to sample realistic objects. In turn, the encoder network at the same time is trained to map the true data distribution to the prior in latent space. To ensure good reconstructions, we introduce an augmented adversarial reconstruction loss. Here we train a discriminator to distinguish two types of pairs: an object with its augmentation and the one with its reconstruction. We show that such adversarial loss compares objects based on the content rather than on the exact match. We experimentally demonstrate that our model generates samples and reconstructions of quality competitive with state-of-the-art on datasets MNIST, CIFAR10, CelebA and achieves good quantitative results on CIFAR10.
BibTeX
@misc{alanov2018pairwise,
  author = {Aibek Alanov and Max Kochurov and Daniil Yashkov and Dmitry P. Vetrov},
  title = {{Pairwise Augmented GANs with Adversarial Reconstruction Loss}},
  year = {2018},
  eprint = {1810.04920},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.04920},
}
Figure