Abstract
Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists of an encoder network, a generator network, and an adversarial network. The encoder network performs style separation by mapping an object photograph to a latent vector. Two halves of the latent vector represent the content and style, respectively. The generator network performs style synthesis by taking a concatenated vector as input. The concatenated vector contains the style half vector of the style target image and the content half vector of the content target image. Once obtaining the images from the generator network, an adversarial network is imposed to generate more photo-realistic images. Experiments on CelebA and UT Zappos 50K datasets demonstrate that the S3-GAN has the capacity of style separation and synthesis simultaneously, and could capture various styles in a single model.
Introduction
In computer vision, style transfer typically involves applying the artistic texture of one image to the structure of another. However, the "dual problem" of style separation is often overlooked. Most existing models are unidirectional: they can synthesize a style but cannot easily extract a reusable representation of style from a new, unseen photograph. Furthermore, many current methods are computationally expensive, requiring hundreds of iterations per image, or are style-specific, necessitating a new model for every style.
The S3-GAN framework addresses these issues by creating a unified, feed-forward model that understands the independent manifold of content and style for specific object categories.
Methodology
The S3-GAN architecture is designed to disentangle an image into its core "content" and "style" components and then recombine them.
The system consists of four main components:
- Encoder (E): Maps a photograph to a latent vector split into two halves: one representing content (e.g., identity and pose) and the other representing style (e.g., skin tone and lighting).
- Generator (G): The inverse of the encoder, which takes a concatenated content-style vector to synthesize a photo-realistic image.
- Discriminator (D): Ensures generated images look like real photographs using a WGAN-GP approach.
- Perceptual Network (P): A pre-trained VGG-19 network used to calculate perceptual losses, ensuring the preservation of spatial structure and textural "Gram matrix" properties.
Experiments and Datasets
The authors evaluated S3-GAN on two category-specific datasets:
- CelebA: Over 200,000 celebrity face images, where the model learns to separate facial structure from attributes like complexion.
- UT Zappos50K: 50,000 catalog shoe images, separating silhouettes from materials and patterns.
Key Results
S3-GAN demonstrates superior realism and efficiency. In quantitative tests using a face detection confidence score, S3-GAN achieved 0.984, outperforming classic optimization-based methods (~0.94-0.96).
Key findings include:
- Efficiency: As a feed-forward network, style transfer is performed in a single pass.
- Diversity: A single model can generate a massive variety of combinations by swapping style vectors while holding content constant.
- Latent Interpolation: Smooth transitions between identities or shoe styles prove the model has learned a continuous, meaningful latent representation.
Conclusion
While S3-GAN is currently category-specific and uses a fixed latent split, it provides a powerful framework for simultaneous style separation and synthesis. Future work may extend this to more complex scenes or universal style transfer across unrelated domains.
Summary updated