Unified Generative Adversarial Networks for Controllable Image-to-Image Translation

Hao Tang, Hong Liu, Nicu Sebe

Abstract

We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fr'echet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks.

Introduction

Image-to-image (I2I) translation has achieved significant milestones with frameworks like Pix2pix and CycleGAN. However, two primary challenges persist: scalability and fine-grained control. Most existing models are domain-specific, requiring $O(n^2)$ generators to translate between $n$ domains, which is computationally inefficient. Furthermore, standard I2I models often struggle with large geometric transformations, such as changing a hand gesture from a "fist" to an "open palm," due to a lack of explicit structural guidance.

This paper introduces a unified GAN framework that addresses these hurdles by conditioning a single generator on "controllable structures"—such as skeletons, keypoints, or semantic maps. This approach allows for precise steering of the generation process across diverse tasks while maintaining a compact model size.

Methodology

The core of the proposed framework is a unified architecture that treats various guidance signals as a generic controllable structure ($C$).

Unified Architecture

The system consists of a single generator and dual discriminators. The generator takes a source image and a target structure as input to produce the translated image. To ensure both visual realism and structural fidelity, the model employs two discriminators: one evaluates the image quality alone, while the other assesses the pair of the generated image and its corresponding target structure.

The proposed unified GAN framework, showing the generator (G) conditioned on a target structure and the discriminator (D) evaluating real/fake triplets.

Novel Loss Functions

To improve performance, the authors introduce three key losses:

  1. Channel-Wise Color Loss: Standard losses often suffer from "channel pollution," where errors in one color channel are masked by others. By calculating L1/L2 losses for R, G, and B channels independently, the model achieves sharper, more color-accurate results.
  2. Structure-Guided Cycle Consistency: This extends the cycle-consistency principle by incorporating structural guides at every step (Source $\rightarrow$ Target $\rightarrow$ Reconstructed Source), ensuring spatial integrity throughout the translation.
  3. Self-Content Preserving Loss: This acts as a regularizer, forcing the generator to maintain the identity and layout of the source image when the input structure already matches the source.

Experiments and Results

The model was evaluated on two challenging tasks: hand gesture-to-gesture translation (using NTU Hand Digit and Creative Senz3D datasets) and cross-view image translation (using Dayton and CVUSA datasets).

Quantitative Performance

The proposed model demonstrates superior efficiency and accuracy. In cross-view tasks, it utilizes approximately 16.9M parameters, significantly fewer than SelectionGAN’s 58.2M. On the Dayton aerial-to-ground task, it achieved a Top-1 Accuracy of 49.12%, outperforming the previous state-of-the-art. Additionally, the authors introduced the Fréchet ResNet Distance (FRD), which measures semantic similarity between individual pairs of real and generated images, providing a more granular evaluation than distribution-based metrics like FID.

Qualitative Comparison

Visual results highlight the model's ability to generate clean boundaries and coherent structures with minimal artifacts.

Qualitative comparison across multiple datasets and baselines, demonstrating the proposed model's superior structural accuracy and visual quality.

Conclusion

This work presents a versatile and efficient GAN framework for controllable image-to-image translation. By abstracting structural guidance and introducing targeted loss functions, the model achieves state-of-the-art results across disparate domains. Future work will focus on reducing the dependency on pre-trained structure extractors and exploring more complex 3D or dynamic structural guides.

Summary updated

Abstract

We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fr\'echet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks. Code is available at https://github.com/Ha0Tang/GestureGAN.
BibTeX
@article{tang2020unified,
  author = {Hao Tang and Hong Liu and Nicu Sebe},
  title = {{Unified Generative Adversarial Networks for Controllable Image-to-Image Translation}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=AmGEnANkGtIk},
}
Figure