Unified Generative Adversarial Networks for Controllable Image-to-Image Translation

Hao Tang, Hong Liu, Nicu Sebe

TL;DR

This paper introduces a unified Generative Adversarial Network (GAN) framework capable of performing controllable image-to-image translation across diverse tasks—such as changing hand gestures or shifting between aerial and ground views. By conditioning a single generator on "controllable structures" (like skeletons or semantic maps) and employing a novel channel-wise color loss, the model achieves state-of-the-art realism and structural accuracy with significantly fewer parameters than task-specific models.

What Problem Does This Paper Address?

Image-to-image (I2I) translation has seen massive success with models like Pix2pix and CycleGAN, but they face two major hurdles:

  1. Scalability: Most frameworks are designed for specific pairs of domains. To translate between $n$ domains, many models require $O(n^2)$ generators, which is computationally prohibitive.
  2. Lack of Fine-Grained Control: Standard I2I models often struggle when the source and target images have large geometric differences (e.g., a hand moving from a "fist" to an "open palm"). They lack a mechanism to "steer" the generation using specific structural guides.

Furthermore, the authors identify a phenomenon called "channel pollution," where standard loss functions allow errors in one color channel to be compensated for by others, leading to unrealistic tints and blurry artifacts in the generated output.

Methods

The authors propose a Unified GAN architecture that treats various forms of guidance—class labels, keypoints, human skeletons, and semantic maps—as a generic "controllable structure" ($C$).

Key Innovations:

  • Unified Generator & Dual Discriminators: A single generator takes a source image and a target structure as input. To ensure the output is both realistic and structurally correct, the authors use two discriminators: one standard (image-only) and one structure-guided (image + structure).
  • Channel-Wise Color Loss: To combat "channel pollution," the model calculates L1/L2 losses for the R, G, and B channels independently. This forces the generator to produce accurate color values for each channel rather than "averaging" them out.
  • Structure-Guided Cycle Consistency: The model employs a "full mapping cycle" (Source $\rightarrow$ Target $\rightarrow$ Reconstructed Source). Unlike the unsupervised CycleGAN, this version uses structural guides at every step to maintain spatial integrity.
  • Self-Content Preserving Loss: This regularizes the generator to act as an identity mapping when the input structure already matches the source image, helping preserve identity and global layout.
  • Frechet ResNet Distance (FRD): The authors introduce a new evaluation metric. While the popular FID (Frechet Inception Distance) measures how well two distributions of images match, FRD measures semantic similarity between individual pairs of real and generated images using a ResNet-50 backbone.

Datasets

The model was evaluated on two distinct types of tasks:

  1. Hand Gesture-to-Gesture:
    • NTU Hand Digit: 10 hand gestures with depth and color data.
  2. Creative Senz3D: Static hand gestures from multiple individuals.
  3. Structural Guide: Hand skeletons extracted via OpenPose.
  4. Cross-View Image Translation:
    • Dayton: Aerial-to-ground and ground-to-aerial street view images.
  5. CVUSA: A similar dataset for cross-view synthesis.
  6. Structural Guide: Semantic segmentation maps generated via RefineNet.

Key Insights and Ideas

  • Structure as a Universal Interface: The most powerful idea here is the abstraction of "structure." By treating a skeleton and a semantic map as the same type of input tensor, the authors prove that a single architecture can generalize across wildly different computer vision tasks.
  • The Importance of Independent Channels: The "aha moment" regarding channel pollution suggests that holistic image losses are often too "forgiving." Forcing the model to get the Red channel right independently of the Blue channel significantly sharpens the results.
  • Semantic vs. Distributional Evaluation: The introduction of FRD highlights a shift in how we evaluate GANs. It’s not just about whether the generated images "look like" the dataset; it's about whether the generated "fist" actually matches the specific ground-truth "fist" semantically.

Key Results

The proposed model consistently outperformed state-of-the-art baselines (like PG$^2$, SelectionGAN, and GestureGAN) in both qualitative and quantitative measures:

  • Efficiency: The model uses significantly fewer parameters. For cross-view tasks, it uses ~16.9M parameters compared to SelectionGAN’s ~58.2M.
  • Accuracy: On the Dayton dataset (aerial-to-ground), the model achieved a Top-1 Accuracy of 49.12%, a significant jump over SelectionGAN's 42.11%.
  • Visual Quality: Qualitative results show much cleaner finger boundaries in gesture tasks and more coherent road/building structures in cross-view tasks, with far fewer "ghosting" artifacts.
  • User Study: In Amazon Mechanical Turk (AMT) perceptual studies, the model "fooled" humans more often than any other baseline, achieving a 29.3% "real" rating on NTU Hand Digit.

Limitations and Future Work

  • Dependency on Pre-trained Extractors: The model relies on external tools (like OpenPose or RefineNet) to provide the structural guides for training. If these extractors fail, the translation quality suffers.
  • Retraining Requirements: While the architecture is unified, the model still needs to be trained on specific task datasets. A truly "zero-shot" unified model that can switch between gestures and cross-view tasks without retraining remains a future goal.
  • Complexity of Structures: The authors suggest that future work could explore even more complex structural guides, such as 3D meshes or dynamic video-based skeletons.

Summary updated

Abstract

We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fréchet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks. Code is available at https://github.com/Ha0Tang/GestureGAN.
BibTeX
@article{tang2020unified,
  author = {Hao Tang and Hong Liu and Nicu Sebe},
  title = {{Unified Generative Adversarial Networks for Controllable Image-to-Image Translation}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=AmGEnANkGtIk},
}

Loading related content...

Figure