Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement

Weihao Xia, Yujiu Yang, Jing-Hao Xue

Abstract

Image-to-image translation has drawn great attention during the past few years. It aims to translate an image in one domain to a given reference image in another domain. Due to its effectiveness and efficiency, many applications can be formulated as image-to-image translation problems. However, three main challenges remain in image-to-image translation: 1) the lack of large amounts of aligned training pairs for different tasks; 2) the ambiguity of multiple possible outputs from a single input image; and 3) the lack of simultaneous training of multiple datasets from different domains within a single network. We also found in experiments that the implicit disentanglement of content and style could lead to unexpect results. In this paper, we propose a unified framework for learning to generate diverse outputs using unpaired training data and allow simultaneous training of multiple datasets from different domains via a single network. Furthermore, we also investigate how to better extract domain supervision information so as to learn better disentangled representations and achieve better image translation. Experiments show that the proposed method outperforms or is comparable with the state-of-the-art methods.

Introduction

Image-to-image (I2I) translation is a fundamental task in computer vision, enabling the transformation of visual data across domains—such as converting summer scenes to winter or sketches to realistic photos. Despite significant progress with models like CycleGAN, existing frameworks often struggle with three key limitations: the requirement for paired training data, the inability to scale efficiently across multiple domains, and the lack of multimodal output diversity.

A critical, often overlooked issue is the "disentanglement trap." Most models rely on implicit learning to separate an image's content (structural features) from its style (domain-specific appearance). This implicit approach often leads to "leaky" representations where style information contaminates content features, resulting in artifacts or loss of structural integrity during translation. To address these challenges, we introduce DCM²IT, a unified framework for unsupervised multi-domain multimodal translation featuring explicit domain-constrained disentanglement.

Methodology: The DCM²IT Framework

The DCM²IT architecture is designed to be both scalable and robust. Unlike pairwise models that require $n(n-1)$ generators for $n$ domains, DCM²IT utilizes a single generator conditioned on domain labels, significantly reducing computational overhead.

Architecture and Translation Pipeline

The framework consists of three primary components: a content encoder ($E^c$), a style encoder ($E^s$), and a generator ($G$). The content encoder extracts domain-invariant features, while the style encoder captures domain-specific attributes.

The DCM²IT framework overview, illustrating the translation process between domains x and y using content and style encoders to produce diverse outputs.

As shown in the figure above, the model can perform both within-domain reconstruction and cross-domain translation. By swapping style codes between images or sampling from a learned distribution, the generator can produce a wide variety of outputs for a single input image.

Explicit Domain-Constrained Disentanglement

The core innovation of DCM²IT is the transition from implicit to explicit disentanglement. By introducing a Domain Style Representation Extractor, the model is forced to learn a shared style distribution for images within the same domain. This explicit constraint prevents the style encoder from "stealing" structural information from the content encoder.

Comparison of disentanglement strategies: (a) standard implicit disentanglement, (b) disentanglement with a Gaussian prior, and (c) the proposed explicit domain-constrained approach.

Distribution Matching via MMD

To facilitate multimodal generation, the style space is mapped to a standard Gaussian distribution. While traditional Variational Autoencoders (VAEs) use KL divergence for this purpose, DCM²IT employs Maximum Mean Discrepancy (MMD). The researchers found that KL divergence was often too restrictive, leading to poor reconstruction quality. MMD provides a more flexible constraint that preserves image quality while enabling diverse style sampling.

Experiments and Results

The framework was rigorously tested on several multi-domain datasets, including Art (Real, Monet, Van Gogh, Ukiyo-e), Weather (Sunny, Cloudy, Snowy, Foggy), and Seasons (Spring, Summer, Autumn, Winter).

Quantitative Performance

DCM²IT demonstrated superior performance compared to state-of-the-art baselines such as StarGAN, MUNIT, and DRIT. On the Season dataset, the model achieved a Fréchet Inception Distance (FID) of 73.47, indicating high image realism, and a Learned Perceptual Image Patch Similarity (LPIPS) score of 0.4810, reflecting significant output diversity.

Human Evaluation and Efficiency

User studies confirmed the model's effectiveness, with DCM²IT's results consistently rated as more realistic than competitors. In a classification task, human observers correctly identified the generated seasons with 65.8% accuracy, outperforming the baseline for real seasonal images (48.9%). Furthermore, the unified architecture proved highly efficient, achieving these results with a single model where baselines like ComboGAN required eight separate generators.

Conclusion

DCM²IT represents a significant advancement in unsupervised image-to-image translation. By implementing explicit domain-constrained disentanglement and leveraging MMD for distribution matching, the framework successfully separates content from style across multiple domains. This results in high-quality, diverse image generation that preserves structural integrity. Future research will focus on extending these principles to temporal domains, such as video-to-video translation.

Summary updated

Abstract

Image-to-image translation has drawn great attention during the past few years. It aims to translate an image in one domain to a given reference image in another domain. Due to its effectiveness and efficiency, many applications can be formulated as image-to-image translation problems. However, three main challenges remain in image-to-image translation: 1) the lack of large amounts of aligned training pairs for different tasks; 2) the ambiguity of multiple possible outputs from a single input image; and 3) the lack of simultaneous training of multiple datasets from different domains within a single network. We also found in experiments that the implicit disentanglement of content and style could lead to unexpect results. In this paper, we propose a unified framework for learning to generate diverse outputs using unpaired training data and allow simultaneous training of multiple datasets from different domains via a single network. Furthermore, we also investigate how to better extract domain supervision information so as to learn better disentangled representations and achieve better image translation. Experiments show that the proposed method outperforms or is comparable with the state-of-the-art methods.
BibTeX
@article{xia2020unsupervised,
  author = {Weihao Xia and Yujiu Yang and Jing-Hao Xue},
  title = {{Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement}},
  journal = {Neural Networks},
  year = {2020},
  doi = {10.1016/j.neunet.2020.07.023},
}
Figure