Asymmetric GAN for Unpaired Image-to-image Translation

Yu Li, Sheng Tang, Rui Zhang, Yongdong Zhang, Jintao Li, Shuicheng Yan

Abstract

Unpaired image-to-image translation aims to model the mapping from one domain to another with unpaired training data. Current works like the well-acknowledged Cycle GAN provide a general solution through modeling injective mappings with a symmetric structure. However, in situations where two domains are asymmetric in complexity—meaning the amount of information between domains is different—these approaches suffer from poor generation quality, mapping ambiguity, and model sensitivity. To address these issues, we propose Asymmetric GAN (AsymGAN). By introducing an auxiliary variable (aux) to learn extra information when transferring from an information-poor domain to an information-rich domain, AsymGAN improves performance in three key ways: it better balances information for higher quality generation, models one-to-many mappings through a controllable aux variable, and increases robustness by decoupling the ill-conditioned relevance of generators. We verify the effectiveness of AsymGAN qualitatively and quantitatively on asymmetric tasks using the Cityscapes and Helen datasets.

Introduction

Unpaired image-to-image (I2I) translation typically assumes a symmetric relationship between two domains. However, many real-world tasks are inherently asymmetric. For instance, a semantic label map (information-poor) contains significantly less visual complexity than a high-resolution photo (information-rich).

Conceptual overview of AsymGAN compared to CycleGAN, highlighting information imbalance, diversity, and robustness.

Standard symmetric models like CycleGAN struggle with these tasks, leading to three primary issues:

  1. Poor Quality: Generators struggle to hallucinate high-complexity details from low-complexity inputs without extra guidance.
  2. Mapping Ambiguity: Symmetric cycles force a one-to-one mapping, whereas translation should often be one-to-many (e.g., one label map corresponding to multiple lighting conditions).
  3. Model Sensitivity: CycleGAN often "hides" information in high-frequency "secret encodings" to satisfy cycle consistency. This makes the model extremely fragile; even a tiny shift or crop of the input image can cause reconstruction to fail spectacularly.

Methods: The Asymmetric GAN Framework

To address these challenges, we propose Asymmetric GAN (AsymGAN). The core innovation is the introduction of an auxiliary variable ($z$) and an encoder ($E$) to explicitly capture and inject the information missing in the "poor" domain.

Detailed architecture of AsymGAN showing training loops and inference modes.

Architecture and Loops

The architecture consists of three main components: a generator $G$ ($X \to Y$), a generator $F$ ($Y, Z \to X$), and an encoder $E$ ($X \to Z$).

  • The Forward Loop ($X \to Y \to X$): A photo $x$ is translated to a label $\hat{y}$ by $G$. Simultaneously, $E$ encodes $x$ into a latent vector $\hat{z}$. The reconstruction $x^*$ is then formed by $F(\hat{y}, \hat{z})$. This ensures $F$ learns to utilize auxiliary information to rebuild complex details.
  • The Backward Loop ($Y \to X \to Y$): To generate a photo from a label $y$, the model samples a random $z$ from a prior distribution (e.g., Gaussian). This enables the generation of diverse outputs for a single input by simply changing $z$.
  • Training Constraints: The latent space $Z$ is regularized using an adversarial loss to follow a standard distribution. For smaller datasets, "Extensive Losses" (including perception and total variation losses) are added to stabilize training.

Key Results and Insights

AsymGAN was evaluated on several asymmetric datasets, including Cityscapes (labels $\leftrightarrow$ photos) and Helen (face masks $\leftrightarrow$ photos).

  • Quantitative Superiority: On the Cityscapes photo $\to$ label task, AsymGAN achieved a 74.9% per-pixel accuracy, significantly outperforming CycleGAN's 58.0%.
  • Robustness: Unlike CycleGAN, which fails when images are slightly cropped or scaled due to its reliance on "hidden" information, AsymGAN remains stable because it decouples generators via the explicit auxiliary variable.
  • Controllable Diversity: Because $z$ is a separate input, users can control the output style. One can either sample a random $z$ for a random style or use an encoded $z$ from a different reference photo to perform a "style swap" while maintaining the structure of the original label map.
  • Visual Quality: Qualitative results show much sharper details, such as distinct lane lines on roads and realistic textures on trees, which competing models often smudge into indistinct blobs.

Conclusion

AsymGAN provides a robust solution for unpaired I2I translation in asymmetric domains. By acknowledging and balancing information differences through an auxiliary variable, the model achieves higher generation quality, enables one-to-many mappings, and significantly improves stability against input variations.

Summary updated

Abstract

Unpaired image-to-image translation problem aims to model the mapping from one domain to another with unpaired training data. Current works like the well-acknowledged Cycle GAN provide a general solution for any two domains through modeling injective mappings with a symmetric structure. While in situations where two domains are asymmetric in complexity, i.e., the amount of information between two domains is different, these approaches pose problems of poor generation quality, mapping ambiguity, and model sensitivity. To address these issues, we propose Asymmetric GAN (AsymGAN) to adapt the asymmetric domains by introducing an auxiliary variable (aux) to learn the extra information for transferring from the information-poor domain to the information-rich domain, which improves the performance of state-of-the-art approaches in the following ways. First, aux better balances the information between two domains which benefits the quality of generation. Second, the imbalance of information commonly leads to mapping ambiguity, where we are able to model one-to-many mappings by tuning aux, and furthermore, our aux is controllable. Third, the training of Cycle GAN can easily make the generator pair sensitive to small disturbances and variations while our model decouples the ill-conditioned relevance of generators by injecting aux during training. We verify the effectiveness of our proposed method both qualitatively and quantitatively on asymmetric situation, label-photo task, on Cityscapes and Helen datasets, and show many applications of asymmetric image translations. In conclusion, our AsymGAN provides a better solution for unpaired image-to-image translation in asymmetric domains.
BibTeX
@article{li2020asymmetric,
  author = {Yu Li and Sheng Tang and Rui Zhang and Yongdong Zhang and Jintao Li and Shuicheng YAN},
  title = {{Asymmetric GAN for Unpaired Image-to-image Translation}},
  journal = {IEEE Transactions on Image Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OiZJNyEV1A},
}
Figure