Style Separation and Synthesis via Generative Adversarial Networks

Rui Zhang, Sheng Tang, Yu Li, Junbo Guo, Yongdong Zhang, Jintao Li, Shuicheng Yan

TL;DR

The S3-GAN (Style Separation and Synthesis GAN) is a framework designed to simultaneously disentangle an image into its core "content" and "style" components and then recombine them to create new images. By mapping images to a split latent space, a single model can perform high-quality style transfer across a wide variety of appearances within a specific object category (like faces or shoes) much faster than traditional iterative methods.

What Problem Does This Paper Address?

In the world of computer vision, "style transfer" usually means taking the artistic texture of one image (like a Van Gogh painting) and applying it to the structure of another. However, the authors identify a "dual problem" that is often overlooked: style separation.

Most existing models are "one-way"—they can synthesize a style but can't easily extract a reusable representation of style from a new, unseen photograph. Furthermore, many current methods are either computationally expensive (requiring hundreds of iterations per image) or are "style-specific," meaning you need to train a completely different model for every new style you want to apply. S3-GAN aims to solve this by creating a unified, feed-forward model that understands the independent "manifold" of content and style for specific object categories.

Methods

The S3-GAN architecture consists of four main components:

  1. Encoder (E): Maps a photograph to a latent vector. Crucially, this vector is split in half: one half represents content (e.g., the identity and pose of a face) and the other half represents style (e.g., skin tone and hair color).
  2. Generator (G): The inverse of the encoder. It takes a concatenated content-style vector and synthesizes a photo-realistic image.
  3. Discriminator (D): A standard GAN component that ensures the generated images look like real photographs from the training set (using a WGAN-GP approach for stability).
  4. Perceptual Network (P): A pre-trained VGG-19 network used to calculate "perceptual losses." It ensures the generated image preserves the spatial structure of the content target and the textural "Gram matrix" of the style target.

The model is trained using a weighted combination of five losses: adversarial loss (realism), content perceptual loss (structure), style perceptual loss (texture), reconstruction loss (ensuring the encoder and generator are true inverses), and total variation loss (smoothness).

Datasets

The authors evaluated the model on two category-specific datasets:

  • CelebA: Over 200,000 celebrity face images. The model learns to separate facial structure/pose from attributes like lighting and complexion.
  • UT Zappos50K: 50,000 catalog shoe images. Here, the model separates the shoe's silhouette and type from its material, color, and pattern.

Key Insights and Ideas

  • Disentanglement via Latent Splitting: The core "aha" moment is the assumption that content and style are independent. By forcing the encoder to pack these into separate halves of a latent vector, the model learns a "style-agnostic" content representation and a "content-agnostic" style representation.
  • The Manifold Assumption: The authors treat object categories as lying on a high-dimensional manifold. Style transfer is essentially "traversing" this manifold by swapping coordinates in the latent space.
  • Reconstruction as an Anchor: Unlike some GANs that only care about the final output looking "real," S3-GAN uses a reconstruction loss. This forces the encoder and generator to be a pair of inverse mappings, which helps the model retain fine-grained details (like the shape of eyes or the stitching on a shoe) that purely adversarial models often lose.

Key Results

  • Superior Realism: In quantitative tests using a face detection "confidence score," S3-GAN outperformed classic methods like Gatys et al. and Johnson et al. S3-GAN achieved a confidence score of 0.984, compared to ~0.94-0.96 for baselines, indicating its outputs are more structurally sound.
  • Efficiency: Because it is a feed-forward network, once trained, it can perform style transfer in a single pass, making it significantly faster than optimization-based methods.
  • Diversity: A single S3-GAN model can generate a massive variety of combinations. By holding a content vector constant and swapping style vectors from different images, the model produces the same person with dozens of different skin tones and lighting conditions.
  • Latent Interpolation: The authors demonstrated that you can smoothly transition between two people's identities or two different shoe styles by interpolating between their latent vectors, proving the model has learned a continuous, meaningful representation of the data.

Limitations and Future Work

  • Category Specificity: The model must be trained on a specific category (e.g., just faces or just shoes). It cannot currently perform "universal" style transfer between unrelated domains (like turning a cat into a toaster).
  • Fixed Latent Split: The 50/50 split of the latent vector for content and style is a heuristic. In reality, the "amount" of information needed for content vs. style might vary by object type.
  • Future Directions: The authors suggest that this framework could be extended to more complex scenes or used as a pre-processing step for other tasks like face recognition or attribute manipulation.

Summary updated

Abstract

Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists of an encoder network, a generator network, and an adversarial network. The encoder network performs style separation by mapping an object photograph to a latent vector. Two halves of the latent vector represent the content and style, respectively. The generator network performs style synthesis by taking a concatenated vector as input. The concatenated vector contains the style half vector of the style target image and the content half vector of the content target image. Once obtaining the images from the generator network, an adversarial network is imposed to generate more photo-realistic images. Experiments on CelebA and UT Zappos 50K datasets demonstrate that the S3-GAN has the capacity of style separation and synthesis simultaneously, and could capture various styles in a single model.
BibTeX
@inproceedings{zhang2018style,
  author = {Rui Zhang and Sheng Tang and Yu Li and Junbo Guo and Yongdong Zhang and Jintao Li and Shuicheng YAN},
  title = {{Style Separation and Synthesis via Generative Adversarial Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=Rp31beqSg-a},
}

Loading related content...

Figure