Multi-crop Contrastive Learning for Unsupervised Image-to-Image Translation

Chen Zhao, Wei-Ling Cai, Zheng Yuan, Cheng-Wei Hu

TL;DR

This paper introduces MCDUT, a novel framework for unsupervised image-to-image translation that enhances contrastive learning by generating higher-quality negative samples through multi-cropping techniques. It also incorporates a domain consistency loss to ensure generated images align with the target domain's style and a Dual Coordinate Attention (DCA) network to improve feature representation.

What Problem Does This Paper Address?

Unsupervised image-to-image translation aims to transform an image from a source domain to a target domain without paired examples (e.g., turning a horse into a zebra). While contrastive learning has shown promise, existing methods often struggle with the quality of "negative" samples used during training. These negatives are typically sampled from the input image itself, leading to an unbalanced distribution and making it harder for the model to learn effectively. Furthermore, previous approaches primarily focus on preserving content consistency, neglecting the crucial aspect of domain consistency – ensuring the generated image not only retains content but also matches the visual style of the target domain.

Methods

The core of MCDUT lies in three key innovations:

  • Multi-cropping Contrastive Learning: To address the issue of low-quality negative samples, MCDUT employs a multi-cropping strategy. It generates multiple views of the input image using both center-cropping (focusing on the foreground) and random-cropping (capturing background details). These diverse views are then used to sample more informative negative examples for contrastive learning, improving the model's ability to distinguish between relevant and irrelevant features.
  • Domain Consistency Loss: Beyond content preservation, MCDUT introduces a novel domain consistency loss. This loss encourages the generated images to be semantically and stylistically similar to real images from the target domain in the embedding space. By projecting features from both generated and real target domain images, this loss ensures the output aligns with the target domain's characteristics.
  • Dual Coordinate Attention (DCA) Network: To further enhance feature representation, MCDUT integrates a Dual Coordinate Attention (DCA) network into its generator. DCA embeds positional information into the channel attention mechanism by combining global average pooling (GAP) and global max pooling (GMP) features. This allows the network to capture both horizontal and vertical global dependencies, enabling it to better focus on relevant features for image translation.

Datasets

The paper evaluates MCDUT on several standard image-to-image translation datasets:

  • Horse → Zebra
  • Cat → Dog
  • Summer → Winter
  • Orange → Apple
  • Van Gogh → Photo

Key Insights and Ideas

  • The Importance of Negative Sample Quality: The paper strongly emphasizes that the quality and diversity of negative samples in contrastive learning are critical for effective unsupervised image-to-image translation. Simply sampling from the input image is insufficient.
  • Bridging Content and Domain Consistency: MCDUT highlights the necessity of jointly optimizing for both content preservation and domain consistency. Neglecting domain consistency can lead to outputs that are content-accurate but stylistically mismatched.
  • Attention for Style: The introduction of the DCA network demonstrates that attention mechanisms, particularly those that capture global spatial dependencies, can significantly improve a generator's ability to learn and apply target domain styles.

Key Results

MCDUT achieves state-of-the-art (SOTA) results across multiple image-to-image translation tasks.

  • Quantitative Improvements: In Table 1, MCDUT consistently outperforms baselines like CycleGAN, CUT, DCLGAN, NEGCUT, and QS-Attn on metrics such as FID (Frechet Inception Distance) and KID (Kernel Inception Distance) across various datasets. For instance, on the Horse → Zebra dataset, MCDUT achieves an FID of 36.3 and KID of 0.385, significantly better than other methods.
  • Qualitative Improvements: Visual comparisons (Figures 5-8) show that MCDUT generates clearer, more detailed, and more natural-looking images compared to existing methods. The generated images exhibit better geometry and texture changes, and a more accurate representation of domain-specific features.
  • Ablation Studies: Ablation studies confirm the effectiveness of each component. The multi-cropping strategy (Table 2) shows optimal performance with one center crop and two random crops. The combination of multi-cropping contrastive learning, domain consistency loss, and the DCA network (Table 3) yields the best overall results.

Limitations and Future Work

  • Failure Cases: While MCDUT performs well, the paper acknowledges some failure cases, such as difficulty in identifying animal fur details in the Cat → Dog dataset.
  • Redundant Information: The ablation study on crop strategies found that using too many random crops (five) led to unsatisfactory results due to redundant information. This suggests a need for careful selection of cropping strategies.
  • Future Research: The authors believe their work will inspire further research in unsupervised image-to-image translation, particularly in exploring more effective attention mechanisms and contrastive learning strategies. The DCA network's effectiveness suggests its potential for broader application in other vision tasks.

Summary updated

Abstract

Recently, unsupervised image-to-image translation methods based on contrastive learning have achieved state-of-the-art results in many tasks. However, in the previous works, the negatives are sampled from the input image itself, which inspires us to design a data augmentation method to improve the quality of the selected negatives. Moreover, the previous methods only preserve the content consistency via patch-wise contrastive learning in the embedding space, which ignores the domain consistency between the generated images and the real images of the target domain. In this paper, we propose a novel unsupervised image-to-image translation framework based on multi-cropping contrastive learning and domain consistency, called MCDUT. Specifically, we obtain the multi-cropping views via the center-cropping and the random-cropping with the aim of further generating the high-quality negative examples. To constrain the embeddings in the deep feature space, we formulate a new domain consistency loss, which encourages the generated images to be close to the real images in the embedding space of the same domain. Furthermore, we present a dual coordinate attention network by embedding positional information into the channel, which called DCA. We employ the DCA network in the design of generator, which makes the generator capture the horizontal and vertical global information of dependency. In many image-to-image translation tasks, our method achieves state-of-the-art results, and the advantages of our method have been proven through extensive comparison experiments and ablation research.
BibTeX
@article{zhao2025multi,
  author = {Chen Zhao and Wei-Ling Cai and Zheng Yuan and Cheng-Wei Hu},
  title = {{Multi-crop Contrastive Learning for Unsupervised Image-to-Image Translation}},
  journal = {Iet Image Process},
  year = {2025},
  doi = {10.1049/ipr2.70006},
}
Figure