Zero-Pair Image to Image Translation using Domain Conditional Normalization

Samarth Shukla, Andrés Romero, Luc Van Gool, Radu Timofte

Introduction

Image-to-image (I2I) translation is the task of converting an image from one representation to another, such as turning a depth map into a semantic segmentation. While supervised methods require paired data and unsupervised methods like CycleGAN use unpaired data, the "zero-pair" problem presents a unique challenge. In this setting, paired data exists between a source domain A and a pivot P, and between P and a target B, but no direct pairs exist between A and B. This paper introduces Domain Conditional Normalization (DCN) to address this problem using a single, efficient shared network.

Methodology

The core innovation is the use of a single encoder-decoder architecture where most weights are shared across all tasks. Domain-specific information is injected exclusively through Conditional Batch Normalization (CBN) layers. The authors propose two primary variants:

DCN-0: Domain-Invariant Latent Space

DCN-0 forces the encoder to produce a "domain-invariant" latent space. It employs a Domain Classifier with a gradient reversal layer to ensure that features from different domains are indistinguishable in the latent space. The decoder then uses the target domain as a condition to generate the output.

DCN: Flexible Latent Space and Pseudo-Pairs

The DCN variant does not enforce latent invariance. Instead, the decoder is conditioned on the pair of domains (e.g., Depth to Semantics). To train the model for unseen mappings, the authors use "pseudo-pairs": they translate a pivot RGB image into both Depth and Semantics using the shared network, then use these results as a synthetic pair to train the direct Depth-to-Semantics link.

Architectural comparison of the DCN-0 and DCN approaches.

Experiments and Datasets

The researchers evaluated their methods on the Synscapes and SceneNet RGB-D datasets. Synscapes is a large-scale synthetic dataset of 25,000 virtual road scenes, providing perfectly aligned RGB, depth, and semantic segmentation maps. This alignment allows for rigorous testing of zero-pair translation by using RGB as the pivot domain.

Sample domains from the Synscapes dataset: Depth, RGB, and Semantic Segmentation.

Results and Discussion

DCN significantly outperforms existing zero-pair methods. On the Synscapes Depth-to-Semantics task, DCN achieved a mean Intersection-over-Union (mIoU) of 54.3, compared to 29.1 for the previous state-of-the-art, M&MNet.

The approach is also highly efficient. DCN uses only ~11.4 million parameters, whereas a cascaded pix2pix model requires ~45.5 million parameters to achieve a lower mIoU of 48.3. Qualitatively, DCN produces sharper and more accurate semantic maps than competing methods, which often suffer from blurriness or misalignment.

Conclusion

Domain Conditional Normalization provides a scalable framework for cross-domain translation without direct paired supervision. By leveraging shared weights and domain-specific normalization, the model achieves high performance with minimal parameter overhead. Future work will focus on closing the remaining performance gap between zero-pair and fully supervised translation.

Summary updated

Abstract

In this paper, we propose an approach based on domain conditional normalization (DCN) for zero-pair image-to-image translation, i.e., translating between two domains which have no paired training data available but each have paired training data with a third domain. We employ a single generator which has an encoder-decoder structure and analyze different implementations of domain conditional normalization to obtain the desired target domain output. The validation benchmark uses RGB-depth pairs and RGB-semantic pairs for training and compares performance for the depth-semantic translation task. The proposed approaches improve in qualitative and quantitative terms over the compared methods, while using much fewer parameters.
BibTeX
@misc{shukla2020zero,
  author = {Samarth Shukla and Andrés Romero and Luc Van Gool and Radu Timofte},
  title = {{Zero-Pair Image to Image Translation using Domain Conditional Normalization}},
  year = {2020},
  eprint = {2011.05680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.05680},
}
Figure