Abstract
Image-to-image translation has drawn great attention during the past few years. It aims to translate an image in one domain to a given reference image in another domain. Due to its effectiveness and efficiency, many applications can be formulated as image-to-image translation problems. However, three main challenges remain in image-to-image translation: 1) the lack of large amounts of aligned training pairs for different tasks; 2) the ambiguity of multiple possible outputs from a single input image; and 3) the lack of simultaneous training of multiple datasets from different domains within a single network. We also found in experiments that the implicit disentanglement of content and style could lead to unexpect results. In this paper, we propose a unified framework for learning to generate diverse outputs using unpaired training data and allow simultaneous training of multiple datasets from different domains via a single network. Furthermore, we also investigate how to better extract domain supervision information so as to learn better disentangled representations and achieve better image translation. Experiments show that the proposed method outperforms or is comparable with the state-of-the-art methods.
Introduction
Image-to-image (I2I) translation is a fundamental task in computer vision, enabling the transformation of visual data across domains—such as converting summer scenes to winter or sketches to realistic photos. Despite significant progress with models like CycleGAN, existing frameworks often struggle with three key limitations: the requirement for paired training data, the inability to scale efficiently across multiple domains, and the lack of multimodal output diversity.
A critical, often overlooked issue is the "disentanglement trap." Most models rely on implicit learning to separate an image's content (structural features) from its style (domain-specific appearance). This implicit approach often leads to "leaky" representations where style information contaminates content features, resulting in artifacts or loss of structural integrity during translation. To address these challenges, we introduce DCM²IT, a unified framework for unsupervised multi-domain multimodal translation featuring explicit domain-constrained disentanglement.
Methodology: The DCM²IT Framework
The DCM²IT architecture is designed to be both scalable and robust. Unlike pairwise models that require $n(n-1)$ generators for $n$ domains, DCM²IT utilizes a single generator conditioned on domain labels, significantly reducing computational overhead.
Architecture and Translation Pipeline
The framework consists of three primary components: a content encoder ($E^c$), a style encoder ($E^s$), and a generator ($G$). The content encoder extracts domain-invariant features, while the style encoder captures domain-specific attributes.
As shown in the figure above, the model can perform both within-domain reconstruction and cross-domain translation. By swapping style codes between images or sampling from a learned distribution, the generator can produce a wide variety of outputs for a single input image.
Explicit Domain-Constrained Disentanglement
The core innovation of DCM²IT is the transition from implicit to explicit disentanglement. By introducing a Domain Style Representation Extractor, the model is forced to learn a shared style distribution for images within the same domain. This explicit constraint prevents the style encoder from "stealing" structural information from the content encoder.
Distribution Matching via MMD
To facilitate multimodal generation, the style space is mapped to a standard Gaussian distribution. While traditional Variational Autoencoders (VAEs) use KL divergence for this purpose, DCM²IT employs Maximum Mean Discrepancy (MMD). The researchers found that KL divergence was often too restrictive, leading to poor reconstruction quality. MMD provides a more flexible constraint that preserves image quality while enabling diverse style sampling.
Experiments and Results
The framework was rigorously tested on several multi-domain datasets, including Art (Real, Monet, Van Gogh, Ukiyo-e), Weather (Sunny, Cloudy, Snowy, Foggy), and Seasons (Spring, Summer, Autumn, Winter).
Quantitative Performance
DCM²IT demonstrated superior performance compared to state-of-the-art baselines such as StarGAN, MUNIT, and DRIT. On the Season dataset, the model achieved a Fréchet Inception Distance (FID) of 73.47, indicating high image realism, and a Learned Perceptual Image Patch Similarity (LPIPS) score of 0.4810, reflecting significant output diversity.
Human Evaluation and Efficiency
User studies confirmed the model's effectiveness, with DCM²IT's results consistently rated as more realistic than competitors. In a classification task, human observers correctly identified the generated seasons with 65.8% accuracy, outperforming the baseline for real seasonal images (48.9%). Furthermore, the unified architecture proved highly efficient, achieving these results with a single model where baselines like ComboGAN required eight separate generators.
Conclusion
DCM²IT represents a significant advancement in unsupervised image-to-image translation. By implementing explicit domain-constrained disentanglement and leveraging MMD for distribution matching, the framework successfully separates content from style across multiple domains. This results in high-quality, diverse image generation that preserves structural integrity. Future research will focus on extending these principles to temporal domains, such as video-to-video translation.
Summary updated