Unsupervised content and style learning for multimodal cross-domain image translation

Zhijie Lin, Jingjing Chen, Xiaolong Ma, Chao Li, Huiming Zhang, Lei ZhaoScientific 2024 · DOI

Abstract

Recently, impressive progress has been made in cross-domain image translation using image generation models pre-trained on massive amounts of data since these pre-trained generative models have strong generative capabilities. However, due to the fact that these pre-trained generative models are usually generated based on the principle of diffusion, there are often the following issues, such as color pattern distortion and the inability to maintain content structure during the translation process. Existing methods fail to meet the requirements of cross-domain image translation, where the shape (content structure) of the image should remain unchanged or undergo minimal change. To address these challenges, we propose an unsupervised content and style learning method for cross-domain image translation. Our approach utilizes self-structure attention loss to preserve content structure and color constraint loss to map the color space of the reference image to the translated image. This effectively separates the image's content structure from its color style, enabling high-quality multi-modal cross-domain image translation. Experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches in metrics such as LPIPS, NDB, JS, IS, and excels in maintaining content structure and color patterns.

Introduction

Cross-domain image translation is a fundamental task in computer vision, enabling transformations such as converting summer landscapes to winter or natural photos into artistic woodblock prints. While modern generative models, particularly those based on diffusion, have shown remarkable capabilities, they frequently struggle with two critical issues: structural distortion and color mode collapse. In many high-stakes applications like film post-production, it is essential that the translated image adopts a new style while keeping its underlying structure—its "bones"—perfectly intact.

Traditional unsupervised methods often rely on cycle-consistency constraints, which are computationally expensive and can limit the diversity of generated results. This paper introduces a novel framework that replaces these constraints with a "Self-Structure Attention Loss" and a statistical color matching approach to achieve high-fidelity, multimodal translation.

Methods

The proposed architecture consists of a content encoder ($E_1$), a style encoder ($E_2$), an AdaPoLIN (Adaptive Pointwise Linear) layer, and a decoder ($G$). The core innovation lies in the disentanglement of content and style through two specific loss functions:

The SSAL-GAN architecture, highlighting the Self-Structure Attention Loss (SSAL) and Color Statistic Loss ($L_{cs}$) used to guide the translation process.
  1. Self-Structure Attention Loss (SSAL): Instead of matching pixels directly, this loss calculates a "self-structure attention map." It measures the distance relationship between local patches and all other structures in the image. By forcing the translated image to share the same attention map as the source, the model preserves global layout and long-range dependencies without being restricted by specific pixel values.
  2. Color Statistical Loss ($L_{cs}$): To capture the reference style without losing color richness, the authors use the Relaxed Earth Mover Distance (REMD) in the semantic feature space. This is supplemented by a statistic matching loss that aligns the mean and covariance of feature vectors, preventing the "washed out" look common in other models.

Notably, the framework is flexible; while it can function as a standard GAN, plugging in a pre-trained Diffusion module (SSAL-Diffusion) provides a significant boost in generative quality by providing structural guidance to the diffusion process.

Experimental Results

The model was evaluated on five diverse datasets: Yosemite (Summer2Winter), Monet2Photo, Ukiyoe2Photo, Photo2Vangogh, and a large-scale Weather dataset containing 60,000 images. SSAL-GAN was compared against established baselines including MUNIT, DRIT, and DSMGAN.

Qualitative comparison on the weather dataset. "Ours" demonstrates superior preservation of the original sunny scene's structure while accurately adopting foggy and snowy styles compared to DRIT and DSMGAN.

Quantitative Performance: On the Weather dataset, the diffusion-based variant (SSAL-Diffusion) achieved a Fréchet Inception Distance (FID) of 96.91, significantly outperforming MUNIT (113.79) and DRIT (105.76). Lower FID scores indicate higher image quality and greater diversity.

Qualitative Performance: Visual results show that the method maintains sharp edges and complex textures—such as the masts of a ship or the intricate branches of a tree—that other models often blur or warp. It also successfully avoids mode collapse, capturing the full color palette of the reference style.

Conclusion and Limitations

By focusing on the relative relationships between parts of an image rather than absolute pixel values, SSAL-GAN achieves state-of-the-art structural consistency in unsupervised translation. However, the model's primary limitation is its strictness; because the structural loss is so effective, it cannot handle tasks that require fundamental shape changes, such as "Cat to Dog" translation. Future work will involve developing "soft" structural constraints to allow for more flexible domain adaptations where the silhouette must shift.

Summary updated

Abstract

Recently, impressive progress has been made in cross-domain image translation using image generation models pre-trained on massive amounts of data since these pre-trained generative models have strong generative capabilities.However, due to the fact that these pre-trained generative models are usually generated based on the principle of diffusion, there are often the following issues, such as color pattern distortion and the inability to maintain content structure during the translation process. Existing methods fail to meet the requirements of cross-domain image translation, where the shape (content structure) of the image should remain unchanged or undergo minimal change. To address these challenges, we propose an unsupervised content and style learning method for cross-domain image translation. Our approach utilizes self-structure attention loss to preserve content structure and color constraint loss to map the color space of the reference image to the translated image. This effectively separates the image's content structure from its color style, enabling high-quality multi-modal cross-domain image translation. Experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches in metrics such as LPIPS, NDB, JS, IS, and excels in maintaining content structure and color patterns.
BibTeX
@inproceedings{lin2024unsupervised,
  author = {Zhijie Lin and Jingjing Chen and Xiaolong Ma and Chao Li and Huiming Zhang and Lei Zhao},
  title = {{Unsupervised content and style learning for multimodal cross-domain image translation}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-80508-w},
}
Figure