AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks

Xudong Mao, Qing Li, Haoran XiearXiv 2017 · OpenReview

Abstract

Recently, several methods based on generative adversarial network (GAN) have been proposed for the task of aligning cross-domain images or learning a joint distribution of cross-domain images. One of the methods is to use conditional GAN for alignment. However, previous attempts of adopting conditional GAN do not perform as well as other methods. In this work we present an approach for improving the capability of the methods which are based on conditional GAN. We evaluate the proposed method on numerous tasks and the experimental results show that it is able to align the cross-domain images successfully in absence of paired samples. Furthermore, we also propose another model which conditions on multiple information such as domain information and label information. Conditioning on domain information and label information, we are able to conduct label propagation from the source domain to the target domain. A 2-step alternating training algorithm is proposed to learn this model.

Introduction

Aligning images across domains—such as mapping a "shoe sketch" to a "realistic shoe photo"—is a classic computer vision challenge. Early approaches required paired datasets (e.g., every sketch must have a corresponding photo), which are expensive and often impossible to collect. While unsupervised methods like CoGAN exist, they rely on weight-sharing between separate GANs. Standard Conditional GANs (cGANs), which use a single model conditioned on a domain vector, theoretically should work for this, but they often fail to maintain semantic consistency. AlignGAN addresses this by strategically choosing which network layers to condition on domain information.

Methodology: Strategic Conditioning

The core of AlignGAN is a refinement of the standard cGAN architecture. The authors propose two fundamental rules for successful cross-domain alignment:

  1. Generator Conditioning: The noise input layer (Z) should not be conditioned with domain vectors. This ensures that the highest-level semantics (the "concept" of the image) remain identical across domains. Instead, domain vectors are injected into subsequent deconvolutional layers to handle domain-specific textures and styles.
  2. Discriminator Conditioning: The image input layer is conditioned with domain vectors. This provides a strong signal to the discriminator, helping it distinguish the domain of an image effectively.
AlignGAN architecture for (a) Discriminator and (b) Generator showing domain vector injection points.

For models conditioning on multiple information sources (domain + label), a 2-step alternating training algorithm is used to ensure convergence. Step 1 focuses on label semantics in the source domain, while Step 2 focuses on alignment across domains. To improve training stability, the authors utilized LSGAN (Least Squares GAN), replacing standard sigmoid cross-entropy loss with a least-squares loss to prevent vanishing gradients.

Experimental Results

AlignGAN was evaluated on diverse benchmarks including Digits (MNIST, USPS), Faces (CelebA), and Objects (Handbags, Shoes). The model demonstrated a remarkable ability to preserve identity and pose across domain shifts.

Qualitative results of gender transformation on the CelebA dataset, demonstrating identity preservation across domains.

Key insights from the results include:

  • Semantic Consistency: By keeping the initial noise layer domain-agnostic, the model finds a shared latent representation for content while later layers learn domain style.
  • Unsupervised Label Propagation: By combining domain and label conditioning, the model can learn what a specific class looks like in a source domain and generate it in an unlabeled target domain.
  • Cross-Category Correlation: The model can learn high-level concepts like "rotation" that transcend specific object types, such as aligning the angles of chairs and cars.

Conclusion

AlignGAN demonstrates that layer-specific conditioning is the key to making cGANs effective for unsupervised cross-domain alignment. While the current implementation requires manual model selection due to training fluctuations and operates at moderate resolutions, it provides a robust framework for learning shared representations across disparate visual domains without the need for paired data.

Summary updated

Abstract

Recently, several methods based on generative adversarial network (GAN) have been proposed for the task of aligning cross-domain images or learning a joint distribution of cross-domain images. One of the methods is to use conditional GAN for alignment. However, previous attempts of adopting conditional GAN do not perform as well as other methods. In this work we present an approach for improving the capability of the methods which are based on conditional GAN. We evaluate the proposed method on numerous tasks and the experimental results show that it is able to align the cross-domain images successfully in absence of paired samples. Furthermore, we also propose another model which conditions on multiple information such as domain information and label information. Conditioning on domain information and label information, we are able to conduct label propagation from the source domain to the target domain. A 2-step alternating training algorithm is proposed to learn this model.
BibTeX
@misc{mao2017aligngan,
  author = {Xudong Mao and Qing Li and Haoran Xie},
  title = {{AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks}},
  year = {2017},
  eprint = {1707.01400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.01400},
}
Figure