Unsupervised Image-to-Image Translation: A Review

Henri Hoyez, Cedric Schockaert, Jason R. Rambach, Bruno Mirbach, Didier Stricker2022 · DOI

TL;DR

This paper provides a comprehensive review of unsupervised image-to-image (UI2I) translation, a field that aims to transform images from one domain to another without requiring paired examples. It categorizes existing UI2I methods, discusses their challenges, and highlights key datasets and evaluation metrics, offering a valuable resource for researchers in computer vision and deep learning.

What Problem Does This Paper Address?

Supervised image-to-image translation, while effective, relies on paired datasets where a source image has a corresponding target image. This requirement is often impractical or impossible to meet, especially for complex transformations or rare domains. Unsupervised image-to-image translation (UI2I) addresses this by using unpaired datasets, but it introduces significant training challenges, often leading to unstable training and suboptimal results. This review aims to survey the landscape of UI2I, detailing its core concepts, the evolution of methods, and the persistent challenges.

Methods

The paper categorizes UI2I methods based on their contributions, including:

  • Architecture Changes: Modifications to existing GAN architectures (e.g., StarGAN, StarGANv2, ASAP-NET) to improve efficiency, stability, and high-resolution translation.
  • Complex Translations: Approaches tackling challenging transformations like geometrical changes (e.g., front-to-profile face translation) or large domain gaps (e.g., selfie-to-anime). This includes methods like ACL-GAN and progressive learning techniques.
  • Data Issues: Strategies to reduce data dependency, such as few-shot learning (e.g., FUNIT, COCO-FUNIT) and techniques for handling limited or unbalanced datasets.
  • Attribute Editing: Methods that allow for more fine-grained control over translations, focusing on specific attribute modifications (e.g., StarGANv2, FacialGAN, HifaFace).
  • Guidance: Techniques that use additional information or constraints (e.g., saliency maps, attention mechanisms, class-aware memory networks) to guide the translation process and improve stability.
  • Disentanglement Learning: Methods that aim to separate content and style representations within an image to achieve more controllable and scalable translations.
  • Contrastive Learning: Approaches that leverage self-supervised learning to optimize learned representations, often by maximizing mutual information between image patches and constraining networks with similarity losses.

The review also details the common loss functions used in UI2I, including adversarial loss, cycle consistency loss, and identity loss, and discusses how their interplay affects training.

Datasets

The paper lists and categorizes a wide array of datasets used in UI2I research, including:

  • Face Datasets: CelebA-HQ, FlickrFace-HQ, FFHQ, CelebAMask-HQ, MeetFace, ClassicTV, Front → Profile Face Dataset.
  • Large Changes in Style or Shape: Oxford-102 (Flowers), DeepFashion, INIT Dataset, Bird Dataset, Google Maps.
  • Segmentation Datasets: Cityscapes, ADE20K, Facade Dataset, SGIT-S/R/C.
  • Outdoor Datasets: Yosemite, Foggy Driving.
  • Classical Datasets: Apple ↔ Orange, Colored MNIST, Artwork.

No new datasets were introduced in this review paper; it serves as a survey of existing benchmarks.

Key Insights and Ideas

  • The "Paired Constraint" is the Bottleneck: The core problem UI2I aims to solve is the impracticality of obtaining paired datasets, which severely limits the scope of supervised translation.
  • Cycle Consistency is a Double-Edged Sword: While crucial for enabling unsupervised learning, cycle consistency can be too restrictive, forcing models to hide information and degrade performance. Newer methods explore alternatives or more nuanced cycle consistency.
  • Training Stability Remains a Major Hurdle: UI2I models, especially GANs, are notoriously difficult to train due to adversarial dynamics and complex loss functions. This has driven research into better regularization, architectural improvements, and guidance mechanisms.
  • Beyond Texture Transfer: Early UI2I focused on simple style changes (e.g., horse to zebra). Modern research tackles complex geometric transformations, large domain gaps, and fine-grained attribute editing.
  • Metrics Matter: The paper highlights the evolution of evaluation metrics from human perception to quantitative measures like FID and LPIPS, emphasizing the need for robust evaluation in UI2I.

Key Results

This paper is a review, not an experimental study, so it does not present new quantitative results. Instead, it summarizes and compares the results reported in the literature. Key findings from the reviewed papers include:

  • Dataset Usage: CityScapes and Horse-to-Zebra are frequently used for semantic translation and general UI2I tasks, respectively. CelebA, CelebA-HQ, and Flickr Face HQ are popular for face translation.
  • Metric Usage: FID is the most prevalent metric for evaluating generation quality, followed by LPIPS, SSIM, and PSNR.
  • Method Performance: The paper presents tables comparing various methods on specific datasets (e.g., Horse-to-Zebra, CityScapes) across different metrics. For instance, on the Horse-to-Zebra dataset, LSeSim [7] is noted for good generation quality based on FID, while QS-Attn [105] shows the best SWD. On CityScapes, TSIT [67] achieved the best class accuracy.
  • Efficiency: NICE-GAN [55] is highlighted as a fast model in terms of time/space complexity.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Evaluation Difficulty: Comparing models across different computing stations is challenging due to variations in hardware and software. The authors suggest finding metrics that account for inference time and memory consumption.
  • Model Complexity: Current deep translation models are parameter-heavy, making them unsuitable for portable devices like smartphones. Research into lightweight and fast architectures is ongoing.
  • Data Dependency: Even unsupervised models require large datasets. While web crawling can generate data, creating comprehensive datasets for every scenario remains difficult.
  • Control Over Generations: While methods like FacialGAN [90] offer guidance, they may not provide complete control, as discriminators can enforce unrealistic constraints (e.g., preventing faces without noses).
  • Dataset Consistency: For datasets like Google Maps, the consistency of map layouts across different regions can lead to evaluation problems. The authors suggest releasing more comprehensive, multi-city datasets.
  • FID Limitations: The paper notes that FID, while widely used, can be influenced by outliers and might not always reflect true generation quality without careful consideration of data selection.

The review concludes by emphasizing the rapid progress in UI2I, highlighting improvements in quality, diversity, efficiency, and the ability to handle complex transformations, while also pointing out areas ripe for further investigation.

Summary updated

BibTeX
@misc{hoyez2022unsupervised,
  author = {Henri Hoyez and Cedric Schockaert and Jason R. Rambach and Bruno Mirbach and Didier Stricker},
  title = {{Unsupervised Image-to-Image Translation: A Review}},
  year = {2022},
  doi = {10.3390/s22218540},
}
Figure