Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros

Introduction

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image. Applications range from style transfer and object transfiguration to photo enhancement. Traditionally, successful models like pix2pix relied on supervised learning using paired datasets—collections where every image in Domain A has a corresponding, perfectly aligned ground-truth image in Domain B.

However, for many real-world tasks, obtaining paired training data is difficult or impossible. For instance, while we have vast collections of Monet paintings and landscape photographs, we do not have photographs of the exact scenes Monet painted. CycleGAN addresses this fundamental limitation by enabling image-to-image translation using unpaired datasets, where two independent collections of images are used without any one-to-one correspondence.

The Problem of Unpaired Translation

In the absence of paired examples, the translation task is highly under-constrained. A standard Generative Adversarial Network (GAN) consists of a generator that learns to map an input to a target domain and a discriminator that learns to distinguish between real and generated images. While an adversarial loss can force the generator to produce images that look like they belong to the target domain, it does not guarantee that the output preserves the structural content of the input. A model could, for example, map a horse to a beautiful zebra that is in a completely different pose or location, satisfying the discriminator but failing the translation task.

The CycleGAN Framework

To solve this, the authors introduce the Cycle Consistency Loss. The core intuition is borrowed from language translation: if one translates a sentence from English to French and then back to English, the result should ideally match the original sentence.

The CycleGAN architecture employs two sets of models simultaneously:

  1. Generator $G$ ($X \to Y$) and Discriminator $D_Y$: Tasked with making images from Domain $X$ look like Domain $Y$.
  2. Generator $F$ ($Y \to X$) and Discriminator $D_X$: Tasked with making images from Domain $Y$ look like Domain $X$.

Cycle Consistency Loss

The model is constrained by two cycles:

  • Forward cycle-consistency: For an image $x$ from domain $X$, the image translation cycle should bring $x$ back to the original image: $x \to G(x) \to F(G(x)) \approx x$.
  • Backward cycle-consistency: Similarly, for an image $y$ from domain $Y$, the cycle should satisfy $y \to F(y) \to G(F(y)) \approx y$.

By minimizing the L1 norm between the original image and its "cycled" version, the model is forced to retain the structural properties of the input. This acts as a powerful regularizer that "pins down" the mapping to something structurally sensible without requiring explicit pairs.

Identity Mapping

In tasks like artistic style transfer, the authors also found utility in an identity loss. This forces the generator to change nothing if it is provided an image that already belongs to the target domain (e.g., $G(y) \approx y$). This helps preserve the original color palette and prevents the model from making unnecessary changes.

Experimental Results

The authors demonstrated CycleGAN's versatility across several diverse datasets, including artistic style transfer (Monet, Van Gogh, and Ukiyo-e vs. photographs), object transfiguration (Horses vs. Zebras), and season transfer (Summer vs. Winter in Yosemite).

Quantitative Evaluation

To measure performance, the researchers used Amazon Mechanical Turk (AMT) for "real vs. fake" perceptual studies. On the Map $\to$ Aerial Photo task, CycleGAN fooled human participants approximately 26.8% of the time. This significantly outperformed previous unpaired methods, which typically achieved less than 3% accuracy.

On the Cityscapes benchmark (labels-to-photo), CycleGAN achieved a per-pixel accuracy of 0.52. While this is lower than the supervised pix2pix model (0.71), it is remarkably high considering the model never saw a single paired example during training.

Qualitative Success

Qualitatively, the model excels at "transfiguring" objects. It can successfully add zebra stripes to a moving horse while maintaining the background and the horse's specific gait. It also performs well at photo enhancement, such as simulating shallow depth-of-field from smartphone photos.

Limitations and Future Work

Despite its success, CycleGAN has notable limitations:

  • Geometric Constraints: The model excels at texture and color changes but struggles with significant geometric transformations. It can change a horse's skin to zebra stripes, but it cannot change a dog's body shape into that of a cat.
  • Dataset Bias: The model is highly sensitive to the contents of the training set. For example, if the zebra dataset only contains wild zebras, the model may attempt to turn a person riding a horse into a "zebra-person."
  • Semantic Ambiguity: Without explicit supervision, the model may occasionally flip labels, such as confusing trees for buildings, if the visual features of the domains overlap significantly.

Conclusion

CycleGAN represents a significant leap in unsupervised computer vision. By introducing the concept of cycle consistency, the authors provided a robust framework for learning mappings between domains where paired data is unavailable. This work has opened the door for a wide range of creative and practical applications in image synthesis and style transfer.

Summary updated

Abstract

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.
BibTeX
@inproceedings{zhu2017unpaired,
  author = {Jun-Yan Zhu and Taesung Park and Phillip Isola and Alexei A. Efros},
  title = {{Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=SJ-4hefu-B},
}
Figure