Image-to-Image Translation: Methods and Applications

Yingxue Pang, Jianxin Lin, Tao Qin, Zhibo Chen

Abstract

Image-to-image translation (I2I) aims to transfer images from a source domain to a target domain while preserving the content representations. I2I has drawn increasing attention and made tremendous progress in recent years because of its wide range of applications in many computer vision and image processing problems, such as image synthesis, segmentation, style transfer, restoration, and pose estimation. In this paper, we provide an overview of the I2I works developed in recent years. We will analyze the key techniques of the existing I2I works and clarify the main progress the community has made. Additionally, we will elaborate on the effect of I2I on the research and industry community and point out remaining challenges in related fields.

Introduction

Image-to-Image (I2I) translation is the task of mapping an image from a source domain to a target domain while maintaining the underlying structural content but altering the visual appearance or style. This field has evolved from basic paired mappings to complex scenarios involving multi-domain transformations and few-shot learning.

Overview of Image-to-Image translation tasks, illustrating the process of generating artistic variations and translating styles between different domains.

The primary challenge in I2I is the diversity of data. While supervised methods rely on perfectly aligned pairs (e.g., a sketch and its corresponding photo), unsupervised methods must learn mappings from unpaired datasets. Modern applications often require "multi-domain" changes—such as simultaneously modifying a subject's age and gender—or "few-shot" learning, where the model adapts to a new style from just a few examples.

Methodology

I2I methods are categorized based on their generative backbones and the nature of the training data:

  • Generative Backbones: The field is dominated by Variational Autoencoders (VAEs), which focus on latent space distributions, and Generative Adversarial Networks (GANs), which use a competitive framework to produce high-fidelity results.
  • Two-Domain Translation:
    • Supervised: Methods like pix2pix use paired data for stable mapping.
  • Unsupervised: CycleGAN introduced cycle-consistency, ensuring that translating from A to B and back to A reconstructs the original image.
  • Contrastive Learning: Newer methods like CUT move away from cycle-consistency, using contrastive learning to ensure the input and output share the same information patches, which is more efficient for large geometric changes.
  • Multi-Domain and Disentanglement: Unified models like StarGAN handle multiple attributes within a single network. A key trend is disentangled representation, where models separate an image into a "content code" (shape) and a "style code" (texture), allowing for diverse, multimodal outputs from a single input.

Benchmarks and Datasets

Performance is typically evaluated on two primary benchmarks:

  1. UT-Zap50K: A dataset of 50,000 shoe images used for testing "edge-to-shoe" translation.
  2. CelebA (CelebFaces Attributes): Containing over 200,000 celebrity images with 40 attribute labels, this is the gold standard for multi-domain attribute manipulation.

Key Insights

  • The Evolution of Consistency: While cycle-consistency was revolutionary, it can be overly restrictive. Modern "one-sided" and attention-guided methods are emerging as more flexible alternatives for tasks with significant shape changes.
  • Disentanglement as a Core Component: Successful modern models (like MUNIT or DRIT++) succeed by explicitly separating domain-invariant content from domain-specific style, enabling a variety of artistic results for a single input.
  • Few-Shot Learning: The field is shifting toward "Few-shot I2I," which uses meta-learning to adapt to new domains with minimal data, mimicking human learning capabilities.

Results

Quantitative comparisons highlight the progress in image quality and diversity:

  • Two-Domain Performance: Among unsupervised methods, CUT (2020) outperformed older staples, achieving a Fréchet Inception Distance (FID) of 50.03, indicating superior image quality.
  • Multi-Domain Performance: StarGANv2 (2020) showed a massive leap over its predecessor, achieving an FID of 40.52 and a significantly higher LPIPS score of 0.397, proving its ability to generate diverse and realistic facial transformations.

Conclusion and Future Work

Despite significant progress, challenges remain in high-resolution synthesis and the trade-off between diversity and physical realism. Future research is expected to explore cross-modality translation, applying I2I principles to translate between images, text, and audio in a unified generative framework.

Summary updated

Abstract

Image-to-image translation (I2I) aims to transfer images from a source domain to a target domain while preserving the content representations. I2I has drawn increasing attention and made tremendous progress in recent years because of its wide range of applications in many computer vision and image processing problems, such as image synthesis, segmentation, style transfer, restoration, and pose estimation. In this paper, we provide an overview of the I2I works developed in recent years. We will analyze the key techniques of the existing I2I works and clarify the main progress the community has made. Additionally, we will elaborate on the effect of I2I on the research and industry community and point out remaining challenges in related fields.
BibTeX
@article{pang2022image,
  author = {Yingxue Pang and Jianxin Lin and Tao Qin and Zhibo Chen},
  title = {{Image-to-Image Translation: Methods and Applications}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3109419},
}
Figure