Tag Disentangled Generative Adversarial Network for Object Image Re-rendering

Chaoyue Wang, Chaohui Wang, Chang Xu, Dacheng TaoIJCAI 2017 · OpenReview

Abstract

In this paper, we propose a principled Tag Disentangled Generative Adversarial Networks (TD-GAN) for re-rendering new images for the object of interest from a single image of it by specifying multiple scene properties (such as viewpoint, illumination, expression, etc.). The whole framework consists of a disentangling network, a generative network, a tag mapping net, and a discriminative network, which are trained jointly based on a given set of images that are completely/partially tagged (i.e., supervised/semi-supervised setting). Given an input image, the disentangling network extracts disentangled and interpretable representations, which are then used to generate images by the generative network. In order to boost the quality of disentangled representations, the tag mapping net is integrated to explore the consistency between the image and its tags. Furthermore, the discriminative network is introduced to implement the adversarial training strategy for generating more realistic images. Experiments on two challenging datasets demonstrate the state-of-the-art performance of the proposed framework in the problem of interest.

Introduction

Re-rendering an object from a single 2D image—such as changing a face's lighting, altering its expression, or rotating a chair to a new viewpoint—is a classic computer vision challenge. Traditionally, this required reconstructing a complex 3D model of the object and projecting it back into 2D, which is computationally expensive and often ill-posed. Recent deep learning methods tackle this by learning "disentangled" representations directly from 2D images. However, previous methods often ignore explicit tag information associated with training images and produce blurry outputs due to reliance on simple reconstruction losses. This paper introduces Tag Disentangled Generative Adversarial Networks (TD-GAN) to address these gaps by explicitly linking image features to tag features and using a Generative Adversarial Network (GAN) to enforce visual realism.

Methodology

TD-GAN is built on four interconnected neural network components designed to learn a shared latent space between visual pixels and semantic tags:

  1. Disentangling Network ($R$): Acts as an encoder that extracts latent, disentangled representations from an input image (e.g., separating identity, viewpoint, and illumination).
  2. Tag Mapping Net ($g$): Translates explicit tag codes (e.g., a one-hot vector for "smile") into the exact same latent space used by $R$.
  3. Generative Network ($G$): Takes the disentangled representations and decodes them back into a reconstructed image.
  4. Discriminative Network ($D$): A standard GAN discriminator that tries to distinguish between genuine images and the re-rendered images produced by $G$.
The TD-GAN architecture, showing the flow from input image through disentangling and generative networks, guided by the tag mapping net.

The key innovation is the enforcement of consistency between image features and tag features. During training, the latent representation extracted by $R$ is penalized if it deviates from the representation generated by $g$ for the corresponding tag. At inference, specific properties can be modified by swapping parts of the latent code with those from the tag mapping net.

Experiments and Results

The authors evaluated TD-GAN on two standard benchmarks: 3D-chairs, containing 86,366 images from 1,393 CAD models across 62 viewpoints, and Multi-PIE, a face dataset featuring 337 subjects under various poses, lighting, and expressions.

TD-GAN achieved state-of-the-art results, significantly outperforming baselines like DC-IGN and XCov. For instance, in illumination tasks on Multi-PIE, TD-GAN achieved a Mean Squared Error (MSE) of $0.10 \times 10^{-2}$, whereas the XCov baseline reached $4.39 \times 10^{-2}$. Qualitative results demonstrate that the inclusion of the GAN discriminator effectively removes the blurriness common in previous auto-encoder-based approaches.

Multi-factor re-rendering results on the Multi-PIE dataset, demonstrating simultaneous control over viewpoint and expression.

Furthermore, TD-GAN supports semi-supervised learning. In tests on the 3D-chairs dataset, a model trained with only 100 tagged samples and 400 untagged samples performed nearly as well as one trained on 500 fully tagged samples (MSE of 0.04379 vs 0.03875), demonstrating the framework's efficiency in data-constrained scenarios.

Conclusion

TD-GAN provides a robust framework for object re-rendering by leveraging explicit tag information and adversarial training. By forcing consistency between visual and semantic representations, it achieves high-quality disentanglement and realistic image generation. While current results rely on discrete tags, future work may extend this framework to continuous scene properties and more complex multi-object environments.

Summary updated

Abstract

In this paper, we propose a principled Tag Disentangled Generative Adversarial Networks (TD-GAN) for re-rendering new images for the object of interest from a single image of it by specifying multiple scene properties (such as viewpoint, illumination, expression, etc.). The whole framework consists of a disentangling network, a generative network, a tag mapping net, and a discriminative network, which are trained jointly based on a given set of images that are completely/partially tagged (i.e., supervised/semi-supervised setting). Given an input image, the disentangling network extracts disentangled and interpretable representations, which are then used to generate images by the generative network. In order to boost the quality of disentangled representations, the tag mapping net is integrated to explore the consistency between the image and its tags. Furthermore, the discriminative network is introduced to implement the adversarial training strategy for generating more realistic images. Experiments on two challenging datasets demonstrate the state-of-the-art performance of the proposed framework in the problem of interest.
BibTeX
@inproceedings{wang2017tag,
  author = {Chaoyue Wang and Chaohui Wang and Chang Xu and Dacheng Tao},
  title = {{Tag Disentangled Generative Adversarial Network for Object Image Re-rendering}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=S1m-iSf_-S},
}
Figure