Abstract
Unpaired image-to-image translation aims to model the mapping from one domain to another with unpaired training data. Current works like the well-acknowledged Cycle GAN provide a general solution through modeling injective mappings with a symmetric structure. However, in situations where two domains are asymmetric in complexity—meaning the amount of information between domains is different—these approaches suffer from poor generation quality, mapping ambiguity, and model sensitivity. To address these issues, we propose Asymmetric GAN (AsymGAN). By introducing an auxiliary variable (aux) to learn extra information when transferring from an information-poor domain to an information-rich domain, AsymGAN improves performance in three key ways: it better balances information for higher quality generation, models one-to-many mappings through a controllable aux variable, and increases robustness by decoupling the ill-conditioned relevance of generators. We verify the effectiveness of AsymGAN qualitatively and quantitatively on asymmetric tasks using the Cityscapes and Helen datasets.
Introduction
Unpaired image-to-image (I2I) translation typically assumes a symmetric relationship between two domains. However, many real-world tasks are inherently asymmetric. For instance, a semantic label map (information-poor) contains significantly less visual complexity than a high-resolution photo (information-rich).
Standard symmetric models like CycleGAN struggle with these tasks, leading to three primary issues:
- Poor Quality: Generators struggle to hallucinate high-complexity details from low-complexity inputs without extra guidance.
- Mapping Ambiguity: Symmetric cycles force a one-to-one mapping, whereas translation should often be one-to-many (e.g., one label map corresponding to multiple lighting conditions).
- Model Sensitivity: CycleGAN often "hides" information in high-frequency "secret encodings" to satisfy cycle consistency. This makes the model extremely fragile; even a tiny shift or crop of the input image can cause reconstruction to fail spectacularly.
Methods: The Asymmetric GAN Framework
To address these challenges, we propose Asymmetric GAN (AsymGAN). The core innovation is the introduction of an auxiliary variable ($z$) and an encoder ($E$) to explicitly capture and inject the information missing in the "poor" domain.
Architecture and Loops
The architecture consists of three main components: a generator $G$ ($X \to Y$), a generator $F$ ($Y, Z \to X$), and an encoder $E$ ($X \to Z$).
- The Forward Loop ($X \to Y \to X$): A photo $x$ is translated to a label $\hat{y}$ by $G$. Simultaneously, $E$ encodes $x$ into a latent vector $\hat{z}$. The reconstruction $x^*$ is then formed by $F(\hat{y}, \hat{z})$. This ensures $F$ learns to utilize auxiliary information to rebuild complex details.
- The Backward Loop ($Y \to X \to Y$): To generate a photo from a label $y$, the model samples a random $z$ from a prior distribution (e.g., Gaussian). This enables the generation of diverse outputs for a single input by simply changing $z$.
- Training Constraints: The latent space $Z$ is regularized using an adversarial loss to follow a standard distribution. For smaller datasets, "Extensive Losses" (including perception and total variation losses) are added to stabilize training.
Key Results and Insights
AsymGAN was evaluated on several asymmetric datasets, including Cityscapes (labels $\leftrightarrow$ photos) and Helen (face masks $\leftrightarrow$ photos).
- Quantitative Superiority: On the Cityscapes photo $\to$ label task, AsymGAN achieved a 74.9% per-pixel accuracy, significantly outperforming CycleGAN's 58.0%.
- Robustness: Unlike CycleGAN, which fails when images are slightly cropped or scaled due to its reliance on "hidden" information, AsymGAN remains stable because it decouples generators via the explicit auxiliary variable.
- Controllable Diversity: Because $z$ is a separate input, users can control the output style. One can either sample a random $z$ for a random style or use an encoded $z$ from a different reference photo to perform a "style swap" while maintaining the structure of the original label map.
- Visual Quality: Qualitative results show much sharper details, such as distinct lane lines on roads and realistic textures on trees, which competing models often smudge into indistinct blobs.
Conclusion
AsymGAN provides a robust solution for unpaired I2I translation in asymmetric domains. By acknowledging and balancing information differences through an auxiliary variable, the model achieves higher generation quality, enables one-to-many mappings, and significantly improves stability against input variations.
Summary updated