Flipped-Adversarial AutoEncoders

Jiyi Zhang, Hung Dang, Hwee Kuan Lee, Ee‐Chien Chang

Abstract

We propose a flipped-Adversarial AutoEncoder (FAAE) that simultaneously trains a generative model G that maps an arbitrary latent code distribution to a data distribution and an encoder E that embodies an "inverse mapping" that encodes a data sample into a latent code vector. Unlike previous hybrid approaches that leverage adversarial training criterion in constructing autoencoders, FAAE minimizes re-encoding errors in the latent space and exploits adversarial criterion in the data space. Experimental evaluations demonstrate that the proposed framework produces sharper reconstructed images while at the same time enabling inference that captures rich semantic representation of data.

Introduction

In the field of generative modeling, researchers have long struggled with a fundamental trade-off between generation quality and inference capability. Standard Generative Adversarial Networks (GANs) are highly effective at producing sharp, realistic images, but they are notoriously difficult to "invert"—it is hard to find the exact latent code that corresponds to a specific real-world image. Conversely, traditional Autoencoders (AEs), such as Variational Autoencoders (VAEs) or Adversarial Autoencoders (AAEs), are designed for inference but typically produce blurry, "smothered" reconstructions. This blurriness is often a byproduct of pixel-wise distance metrics like the L2 norm, which struggle to capture high-frequency details.

Existing hybrid approaches like BiGAN attempt to bridge this gap by matching joint distributions, but they often fail to produce reconstructions that maintain visual fidelity to the original input. The Flipped-Adversarial AutoEncoder (FAAE) aims to overcome these hurdles by creating a framework that functions as both a high-quality generator and a semantically accurate encoder.

The Flipped-Adversarial AutoEncoder (FAAE) Framework

The core innovation of the FAAE is a structural rearrangement of the training pipeline, specifically contrasting it with the standard Adversarial AutoEncoder (AAE). In a traditional AAE, the discriminator operates in the latent space to ensure the encoder's output matches a chosen prior distribution. The FAAE "flips" this structure: the discriminator is moved to the data space, forcing the generator to produce realistic-looking images.

The proposed FAAE architecture. A latent code $z$ is passed through the generator $G_\theta$ to produce $\hat{x}$, which is then re-encoded by $E_\phi$ into $\hat{z}$. The discriminator $D_w$ operates in the data space to distinguish between real samples $x$ and generated samples $\hat{x}$.

Re-encoding Error and Pipeline

Instead of minimizing the pixel-level distance between an input image and its reconstruction, the FAAE minimizes a re-encoding error in the latent space. The training process follows a specific sequence:

  1. Generation: A latent code $z$ is fed into the Generator ($G_\theta$) to create a synthetic image $\hat{x}$.
  2. Inference: This generated image $\hat{x}$ is then fed into the Encoder ($E_\phi$) to produce a re-encoded latent vector $\hat{z}$.
  3. Optimization: The model minimizes the distance between the original latent code $z$ and the re-encoded vector $\hat{z}$.
  4. Adversarial Training: Simultaneously, a Discriminator ($D_w$) compares the generated image $\hat{x}$ against real data samples $x$.

By focusing on the latent space for error measurement, the model prioritizes semantic consistency—ensuring the underlying meaning of the data is preserved—while the data-space discriminator ensures visual sharpness.

Experimental Results

The authors evaluated the FAAE on three standard computer vision benchmarks: CIFAR-10, CelebA (resized to 32x32), and a subset of ImageNet (resized to 64x64).

Visual Quality and Sharpness

Experimental results demonstrate that the FAAE produces significantly sharper reconstructions than standard AAEs across all datasets. By utilizing an adversarial loss in the data space, the model avoids the blurriness common in VAEs and AAEs, as it is forced to match the distribution of real images, including sharp edges and textures.

Qualitative comparison of image reconstructions on CIFAR-10 and CelebA. The FAAE (bottom row of each set) produces significantly sharper and more detailed images than the standard AAE (middle row).

Semantic Consistency and "Frontalization"

Beyond visual sharpness, the FAAE captures rich semantic representations. A fascinating observation on the CelebA dataset was the model's tendency toward "semantic frontalization." Because the latent space is trained on the average semantic representation (which is predominantly frontal in CelebA), the model often "corrects" side-profile faces into frontal views while maintaining the subject's identity. This suggests that the latent space effectively captures the "essence" of an image rather than just its pixel-perfect arrangement.

Conclusion and Future Work

The Flipped-Adversarial AutoEncoder successfully addresses the trade-off between image sharpness and reconstruction accuracy. By shifting the adversarial focus to the data space and minimizing error in the latent space, it achieves high-quality generation alongside semantically consistent inference. While the model shows great promise, scaling to high-resolution data remains a challenge, as visible artifacts still appear in complex datasets like ImageNet. Future research may explore the utility of this bidirectional mapping for security applications, such as studying membership inference and adversarial robustness.

Summary updated

Abstract

We propose a flipped-Adversarial AutoEncoder (FAAE) that simultaneously trains a generative model G that maps an arbitrary latent code distribution to a data distribution and an encoder E that embodies an "inverse mapping" that encodes a data sample into a latent code vector. Unlike previous hybrid approaches that leverage adversarial training criterion in constructing autoencoders, FAAE minimizes re-encoding errors in the latent space and exploits adversarial criterion in the data space. Experimental evaluations demonstrate that the proposed framework produces sharper reconstructed images while at the same time enabling inference that captures rich semantic representation of data.
BibTeX
@misc{zhang2018flipped,
  author = {Jiyi Zhang and Hung Dang and Hwee Kuan Lee and Ee‐Chien Chang},
  title = {{Flipped-Adversarial AutoEncoders}},
  year = {2018},
  eprint = {1802.04504},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.04504},
}
Figure