Learning Inverse Mappings with Adversarial Criterion

Jiyi Zhang, Hung Dang, Hwee Kuan Lee, Ee-Chien Chang

Abstract

Generative modeling often faces a trade-off between the visual quality of samples and the ability to learn a reliable inverse mapping from data to latent space. While Generative Adversarial Networks (GANs) produce sharp images without an inherent encoder, Autoencoders (AEs) provide bidirectional mapping but often suffer from blurry outputs. This paper presents the Flipped-Adversarial AutoEncoder (F-AAE), which "flips" the traditional Adversarial Autoencoder (AAE) structure. By placing the adversarial discriminator in the data space and minimizing re-encoding error in the latent space, F-AAE achieves high-fidelity image generation alongside accurate semantic reconstruction.

Introduction

A primary goal in representation learning is to develop models that can both generate realistic data and map that data into a meaningful latent space. Standard GANs excel at the former but lack an encoder, making it difficult to find the latent code $z$ for a given real image $x$. Conversely, Variational Autoencoders (VAEs) and AAEs provide an encoder-decoder pair but often produce blurry images due to the use of pixel-wise reconstruction losses (e.g., L2 or L1 distance).

Previous attempts like BiGAN or ALI attempt to learn these mappings simultaneously but often struggle with reconstruction fidelity. The F-AAE addresses this by rethinking where adversarial and reconstruction constraints are applied to optimize both generation and inversion.

The Flipped-Adversarial AutoEncoder (F-AAE)

The F-AAE architecture consists of three components: a Generator ($G_\theta$), an Encoder ($E_\phi$), and a Discriminator ($D_w$). The "flipped" nature of the model refers to its departure from the standard AAE design.

In a standard AAE (Figure 2), the discriminator ensures the latent distribution matches a prior, while the generator minimizes pixel-wise reconstruction error, which often leads to blurring.

The standard Adversarial Autoencoder (AAE) architecture for comparison, where the discriminator operates on the latent space.

F-AAE introduces two key modifications:

  1. Adversarial Criterion in Data Space: The discriminator $D_w$ distinguishes between real images $x$ and generated images $\hat{x} = G_\theta(z)$. This forces $G_\theta$ to produce sharp, realistic samples, effectively acting as a GAN.
  2. Latent Re-encoding Error: Instead of comparing pixels, the model minimizes the distance between a sampled latent vector $z$ and its re-encoded version $\hat{z} = E_\phi(G_\theta(z))$. This "re-encoding error" ensures that the encoder learns to invert the generator without the blurring effects of pixel-level penalties.

Experimental Results

The model was evaluated on CIFAR-10, CelebA, and ImageNet datasets.

Reconstruction and Generation Quality

F-AAE demonstrates a significant improvement in reconstruction quality over AAE and BiGAN. While AAE reconstructions are typically blurry, F-AAE maintains high-frequency details. Compared to BiGAN, which often fails to capture the correct semantic category in complex datasets like ImageNet (e.g., failing to reconstruct specific animal species), F-AAE provides more stable and accurate reconstructions.

Semantic Latent Space

A key insight is that F-AAE prioritizes semantic consistency. On the CelebA dataset, the model occasionally exhibits a "frontal face bias." When reconstructing side-profile images, the model may generate a frontal view that retains the person's identity. This suggests the latent space captures high-level concepts (e.g., identity) rather than just low-level pixel arrangements. Furthermore, linear interpolation in the latent space produces smooth transitions between distinct identities, confirming that the model learns a continuous and well-structured manifold.

Conclusion

The Flipped-Adversarial AutoEncoder provides a robust framework for learning bidirectional mappings. By moving the adversarial loss to the data space and the reconstruction loss to the latent space, it sidesteps the blurriness inherent in traditional autoencoders while maintaining the generative power of GANs. Future work involves exploring the utility of this mapping for downstream tasks such as data augmentation and security analysis.

Summary updated

Abstract

We propose a flipped-Adversarial AutoEncoder (FAAE) that simultaneously trains a generative model G that maps an arbitrary latent code distribution to a data distribution and an encoder E that embodies an "inverse mapping" that encodes a data sample into a latent code vector. Unlike previous hybrid approaches that leverage adversarial training criterion in constructing autoencoders, FAAE minimizes re-encoding errors in the latent space and exploits adversarial criterion in the data space. Experimental evaluations demonstrate that the proposed framework produces sharper reconstructed images while at the same time enabling inference that captures rich semantic representation of data.
BibTeX
@misc{zhang2018learning,
  author = {Jiyi Zhang and Hung Dang and Hwee Kuan Lee and Ee-Chien Chang},
  title = {{Learning Inverse Mappings with Adversarial Criterion}},
  year = {2018},
  eprint = {1802.04504},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.04504},
}
Figure