Unsupervised multi-domain multimodal image-to-image translation with explicit domain-constrained disentanglement

Weihao Xia, Yujiu Yang, Jing-Hao Xue

In the world of computer vision, Image-to-Image (I2I) translation is the magic behind turning a summer landscape into a winter wonderland or a smartphone photo into a Van Gogh masterpiece. While we’ve seen incredible progress with models like CycleGAN, most existing tools struggle when asked to do three things at once: work without paired training data, handle more than two domains (e.g., switching between four seasons), and produce diverse, "multimodal" results for a single input.

A team of researchers from Tsinghua University and University College London recently published a paper titled "Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement" that tackles these hurdles head-on with a framework they call DCM²IT.

TL;DR

DCM²IT is a unified framework that performs unsupervised image translation across multiple domains simultaneously while generating diverse, high-quality outputs. By using explicit domain-constrained disentanglement, it successfully separates an image's "content" from its "style," ensuring that structural details remain intact even during radical stylistic shifts.

What Problem Does This Paper Address?

Most I2I models suffer from three main limitations:

  1. Data Scarcity: They often require "aligned pairs" (e.g., the exact same street corner photographed in both summer and winter), which are rare and expensive to collect.
  2. Scalability: To translate between $n$ domains, many models require training $n(n-1)$ separate generators.
  3. The Disentanglement Trap: This is the most subtle issue. Ideally, a model should separate content (the shape of a mountain) from style (the snow on it). However, "implicit" learning often confuses the two. For example, a model trying to de-blur a face might accidentally change the shape of the person's jaw because it hasn't truly disentangled the "blur" style from the "face" content.

Methods

The authors introduce DCM²IT (Unsupervised Multi-domain Multimodal Image-to-image Translation), which relies on a few clever architectural choices:

  • Disentangled Representation: The model uses a content encoder to extract domain-invariant features (the "what") and a style encoder to extract domain-specific features (the "how").
  • Explicit Domain Supervision: Unlike previous models that hope the network figures out style on its own, the authors pre-train a Domain Style Representation Extractor. This forces the model to learn the "essence" of a domain (e.g., what makes "Monet" look like "Monet") by exchanging style codes between images of the same domain.
  • MMD over KL Divergence: To ensure the style codes follow a standard distribution (allowing for random sampling of new styles), they use Maximum Mean Discrepancy (MMD). They found that the traditional KL divergence was often too restrictive, leading to poor reconstructions.
  • Unified Architecture: A single generator handles all domains by taking a domain label as input, significantly reducing the computational footprint compared to pairwise models.

Datasets

The researchers evaluated DCM²IT on several diverse multi-domain datasets:

  • Art: 4 domains (Real photos, Monet, Van Gogh, Ukiyo-e).
  • Weather: 4 domains (Sunny, Cloudy, Snowy, Foggy) from the Image2Weather dataset.
  • Season: 4 domains (Spring, Summer, Autumn, Winter) using 6,000 images of the Alps.
  • Edges $\to$ Shoes/Handbags: Standard benchmarks used specifically to test the model's ability to generate diverse outputs from simple sketches.

Key Insights and Ideas

  • The "Aha" Moment on Disentanglement: The researchers proved that implicit disentanglement is often a "leaky" abstraction. By adding an explicit constraint—forcing images from the same domain to share a "mean" style—they prevented the style encoder from "stealing" content information.
  • Style as a Distribution: By matching the style space to a Gaussian distribution using MMD, the model doesn't just swap styles between two images; it can sample new styles from thin air, allowing for infinite variations of "Autumn" for a single "Spring" photo.
  • Preserving Structure: In tasks like image de-blurring, their method preserved facial contours much better than state-of-the-art competitors (like DRIT), proving that their disentanglement was superior.

Key Results

DCM²IT outperformed or matched state-of-the-art methods like StarGAN, MUNIT, and DRIT across several metrics:

  • Quality and Diversity: On the Season dataset, DCM²IT achieved the lowest FID score (73.47), indicating the highest image realism, and the highest LPIPS score (0.4810), indicating the greatest diversity in generated outputs.
  • Human Perception: In user studies, DCM²IT's results were consistently rated as more realistic. Interestingly, while real images of seasons are often hard for humans to tell apart (48.9% accuracy), DCM²IT’s generated seasons were so distinct they achieved a 65.8% classification accuracy.
  • Efficiency: It achieved these results using a single model for all domains, whereas baselines like ComboGAN required 8 generators and 4 discriminators for the same task.

Limitations and Future Work

While DCM²IT is a major step forward, the authors note a few areas for improvement:

  • Domain Ambiguity: Some domains are naturally overlapping (e.g., a late Spring day looks a lot like early Summer). The model sometimes struggles with these "edge cases" where the domain definition itself is fuzzy.
  • Complexity of "Essence": While the model captures the general style well, extremely complex structural changes (like adding a specific type of building architecture) remain a challenge for unsupervised methods.
  • Future Directions: The authors suggest exploring how this explicit disentanglement could be applied to video-to-video translation, where temporal consistency adds another layer of difficulty.

Summary updated

Abstract

Image-to-image translation has drawn great attention during the past few years. It aims to translate an image in one domain to a given reference image in another domain. Due to its effectiveness and efficiency, many applications can be formulated as image-to-image translation problems. However, three main challenges remain in image-to-image translation: 1) the lack of large amounts of aligned training pairs for different tasks; 2) the ambiguity of multiple possible outputs from a single input image; and 3) the lack of simultaneous training of multiple datasets from different domains within a single network. We also found in experiments that the implicit disentanglement of content and style could lead to unexpect results. In this paper, we propose a unified framework for learning to generate diverse outputs using unpaired training data and allow simultaneous training of multiple datasets from different domains via a single network. Furthermore, we also investigate how to better extract domain supervision information so as to learn better disentangled representations and achieve better image translation. Experiments show that the proposed method outperforms or is comparable with the state-of-the-art methods.
BibTeX
@article{xia2020unsupervised,
  author = {Weihao Xia and Yujiu Yang and Jing-Hao Xue},
  title = {{Unsupervised multi-domain multimodal image-to-image translation with explicit domain-constrained disentanglement}},
  journal = {Neural Networks},
  year = {2020},
  doi = {10.1016/j.neunet.2020.07.023},
}

Loading related content...

Figure