Style Separation and Synthesis via Generative Adversarial Networks

Rui Zhang, Sheng Tang, Yu Li, Junbo Guo, Yongdong Zhang, Jintao Li, Shuicheng Yan

Abstract

Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists of an encoder network, a generator network, and an adversarial network. The encoder network performs style separation by mapping an object photograph to a latent vector. Two halves of the latent vector represent the content and style, respectively. The generator network performs style synthesis by taking a concatenated vector as input. The concatenated vector contains the style half vector of the style target image and the content half vector of the content target image. Once obtaining the images from the generator network, an adversarial network is imposed to generate more photo-realistic images. Experiments on CelebA and UT Zappos 50K datasets demonstrate that the S3-GAN has the capacity of style separation and synthesis simultaneously, and could capture various styles in a single model.

Introduction

In computer vision, style transfer typically involves applying the artistic texture of one image to the structure of another. However, the "dual problem" of style separation is often overlooked. Most existing models are unidirectional: they can synthesize a style but cannot easily extract a reusable representation of style from a new, unseen photograph. Furthermore, many current methods are computationally expensive, requiring hundreds of iterations per image, or are style-specific, necessitating a new model for every style.

The S3-GAN framework addresses these issues by creating a unified, feed-forward model that understands the independent manifold of content and style for specific object categories.

Methodology

The S3-GAN architecture is designed to disentangle an image into its core "content" and "style" components and then recombine them.

Overview of the S3-GAN framework for style separation and synthesis.

The system consists of four main components:

  1. Encoder (E): Maps a photograph to a latent vector split into two halves: one representing content (e.g., identity and pose) and the other representing style (e.g., skin tone and lighting).
  2. Generator (G): The inverse of the encoder, which takes a concatenated content-style vector to synthesize a photo-realistic image.
  3. Discriminator (D): Ensures generated images look like real photographs using a WGAN-GP approach.
  4. Perceptual Network (P): A pre-trained VGG-19 network used to calculate perceptual losses, ensuring the preservation of spatial structure and textural "Gram matrix" properties.

Experiments and Datasets

The authors evaluated S3-GAN on two category-specific datasets:

  • CelebA: Over 200,000 celebrity face images, where the model learns to separate facial structure from attributes like complexion.
  • UT Zappos50K: 50,000 catalog shoe images, separating silhouettes from materials and patterns.

Key Results

S3-GAN demonstrates superior realism and efficiency. In quantitative tests using a face detection confidence score, S3-GAN achieved 0.984, outperforming classic optimization-based methods (~0.94-0.96).

Synthesis results on CelebA dataset with quantitative similarity metrics.

Key findings include:

  • Efficiency: As a feed-forward network, style transfer is performed in a single pass.
  • Diversity: A single model can generate a massive variety of combinations by swapping style vectors while holding content constant.
  • Latent Interpolation: Smooth transitions between identities or shoe styles prove the model has learned a continuous, meaningful latent representation.

Conclusion

While S3-GAN is currently category-specific and uses a fixed latent split, it provides a powerful framework for simultaneous style separation and synthesis. Future work may extend this to more complex scenes or universal style transfer across unrelated domains.

Summary updated

Abstract

Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists of an encoder network, a generator network, and an adversarial network. The encoder network performs style separation by mapping an object photograph to a latent vector. Two halves of the latent vector represent the content and style, respectively. The generator network performs style synthesis by taking a concatenated vector as input. The concatenated vector contains the style half vector of the style target image and the content half vector of the content target image. Once obtaining the images from the generator network, an adversarial network is imposed to generate more photo-realistic images. Experiments on CelebA and UT Zappos 50K datasets demonstrate that the S3-GAN has the capacity of style separation and synthesis simultaneously, and could capture various styles in a single model.
BibTeX
@inproceedings{zhang2018style,
  author = {Rui Zhang and Sheng Tang and Yu Li and Junbo Guo and Yongdong Zhang and Jintao Li and Shuicheng YAN},
  title = {{Style Separation and Synthesis via Generative Adversarial Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=Rp31beqSg-a},
}
Figure