TL;DR
Asymmetric GAN (AsymGAN) improves unpaired image-to-image translation by addressing the information imbalance between "rich" domains (like photos) and "poor" domains (like label maps). By introducing an auxiliary variable and an encoder to explicitly capture and inject missing information, the model achieves higher generation quality, enables one-to-many diverse outputs, and significantly increases robustness against small input perturbations.
What Problem Does This Paper Address?
Unpaired image-to-image (I2I) translation, popularized by CycleGAN, typically assumes a symmetric relationship between two domains. For example, a horse and a zebra contain roughly the same amount of visual information. However, many real-world tasks are asymmetric: a semantic label map (domain $Y$) contains far less information than a high-resolution photo (domain $X$).
When standard symmetric models like CycleGAN tackle these tasks, three problems arise:
- Poor Quality: It is mathematically difficult to force a generator to create high-complexity details from low-complexity inputs without extra guidance.
- Mapping Ambiguity: Translation should be one-to-many (one label map could correspond to many different-looking photos), but symmetric cycles force a one-to-one mapping.
- Model Sensitivity: To satisfy the "cycle consistency" requirement (reconstructing the original image perfectly), CycleGAN often "hides" information from the rich domain into the poor domain using invisible, high-frequency "secret encodings." This makes the model extremely fragile; even a tiny shift or crop of the image can cause the reconstruction to fail spectacularly.
Methods
The authors propose Asymmetric GAN (AsymGAN), which breaks the symmetry of the standard cycle. The architecture consists of three main components: a generator $G$ ($X \to Y$), a generator $F$ ($Y, Z \to X$), and an encoder $E$ ($X \to Z$).
- The Auxiliary Variable ($aux$ or $z$): Instead of $F$ mapping directly from $Y$ to $X$, it now takes an additional latent vector $z$. This $z$ is designed to capture the "lost" information—the details present in the photo but absent in the label.
- The Forward Loop ($X \to Y \to X$): A photo $x$ is translated to a label $\hat{y}$ by $G$. Simultaneously, $E$ encodes $x$ into a latent vector $\hat{z}$. The reconstruction $x^*$ is then formed by $F(\hat{y}, \hat{z})$. This ensures $F$ learns to use the auxiliary information to rebuild the complex photo.
- The Backward Loop ($Y \to X \to Y$): To generate a photo from a label $y$, the model samples a random $z$ from a prior distribution (like a Gaussian). This allows the model to produce diverse outputs for the same input by simply changing $z$.
- Training Constraints: The latent space $Z$ is regularized using an adversarial loss to follow a standard distribution, preventing the model from simply ignoring $z$ or using it in trivial ways. For smaller datasets, the authors add "Extensive Losses," including perception and total variation (TV) losses, to stabilize training.
Datasets
The model was evaluated on several asymmetric and multi-modal datasets:
- Cityscapes: Semantic labels $\leftrightarrow$ street photos.
- Helen: Face parsing masks $\leftrightarrow$ face photos.
- Edges2shoes: Edge maps $\leftrightarrow$ product photos.
- Additional Applications: Facades, Map $\leftrightarrow$ Aerial photo, Night $\leftrightarrow$ Day, Bald $\leftrightarrow$ Hair, and various Style Transfer sets (Monet, Van Gogh, Ukiyo-e).
Key Insights and Ideas
- Explicit vs. Implicit Encoding: The most profound insight is the "Model Sensitivity" analysis. The authors demonstrate that CycleGAN satisfies its cycle loss by hiding information in the low-order bits of pixels. AsymGAN makes this information flow explicit through the auxiliary variable, which "decouples" the generators and leads to a much more stable and robust model.
- Controllable Diversity: Because $z$ is a separate input, users can control the output. You can either sample a random $z$ for a random style or use an encoded $z$ from a different reference photo to perform a sort of "style swap" while keeping the structure of the original label map.
- Information Balancing: By acknowledging that one domain is "poorer" than the other, the architecture provides a "container" (the $aux$ variable) for the missing complexity, rather than forcing the network to hallucinate it from nothing.
Key Results
- Quantitative Superiority: On the Cityscapes photo $\to$ label task, AsymGAN achieved a 74.9% per-pixel accuracy, significantly higher than CycleGAN's 58.0%.
- Robustness: The paper shows that while CycleGAN's reconstructions fail when an image is cropped by just a few pixels or slightly scaled, AsymGAN remains stable.
- Human Evaluation: In a Mean Opinion Score (MOS) survey, AsymGAN outperformed CycleGAN in categories of "Correctness," "Realness," and "Richness."
- Visual Quality: Qualitative results show much sharper details in generated photos, such as distinct lane lines on roads and realistic textures on trees, which competing models often smudge into gray or green blobs.
Limitations and Future Work
- Failure Cases: The authors note that in complex portrait-to-photo translations (like Ukiyo-e paintings), the model sometimes fails to perfectly map human features if the domain gap is too extreme.
- Hyperparameter Sensitivity: The "Extensive Losses" version requires tuning several $\lambda$ weights, which might be difficult for new datasets.
- Future Directions: The authors suggest that exploring different ways to combine $z$ in the generator (like Conditional Instance Normalization) could further improve the diversity and quality of the results.
Summary updated