SSCGAN: Facial Attribute Editing via Style Skip Connections

Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji

Introduction

Facial attribute editing—tasks such as altering age, gender, or adding accessories like eyeglasses—has traditionally relied on encoder-decoder architectures. Most existing frameworks treat these attributes as local labels, often concatenating a one-hot attribute vector directly onto feature maps. However, this approach faces two significant hurdles: the neglect of global context and the "detail dilemma."

The authors of SSCGAN argue that attributes are not merely local patches but influence the global "style" and statistical distribution of an image. Furthermore, standard skip connections, while useful for preserving identity, often "leak" source attributes into the output, hindering effective editing. Conversely, removing these connections leads to a loss of fine spatial details. SSCGAN addresses these issues by decoupling style manipulation from spatial detail preservation.

Methodology: The SSCGAN Architecture

The core innovation of this work is the Style Skip Connection GAN (SSCGAN). It utilizes a symmetric 6-stage encoder-decoder structure enhanced by two novel modules: Style Skip Connections (SSC) and Spatial Information Transfer (SIT).

The SSCGAN architecture featuring Style Skip Connections (SSC) and Spatial Information Transfer (SIT) modules.

Style Skip Connections (SSC)

Instead of transferring raw feature maps, the SSC module extracts channel-wise mean and variance—mathematical representations of "style"—from the encoder. The model calculates a "style residual" based on the difference between source and target attributes. This residual is then injected into the decoder via Adaptive Instance Normalization (AdaIN). This mechanism allows the network to modify the global characteristics of the face at multiple resolutions, from coarse $8 \times 8$ structures to fine $128 \times 128$ textures.

Spatial Information Transfer (SIT)

To compensate for the spatial invariance of style vectors, the SIT module preserves structural integrity. It extracts spatial details from the encoder and uses an attribute-difference-guided attention mechanism to fuse them into the decoder. This ensures that while the style (e.g., hair color) changes, the underlying structure (e.g., individual hair strands) remains sharp and consistent with the original identity.

Key Insights: Style as a Discriminative Feature

A pivotal finding of the research is that global style statistics are highly discriminative for facial attributes. By applying t-SNE visualization to the style features extracted at different stages, the authors demonstrated that these features naturally cluster according to specific attributes.

t-SNE visualization of style features across six stages for "Eyeglasses" and "Male" attributes, demonstrating attribute clustering at various network depths.

The visualization reveals an attribute hierarchy: different features become distinct at different layers. For instance, "Eyeglasses" are most discriminative at Stage 4, whereas gender-based features ("Male") are most distinct at Stage 3. By implementing skip connections across all six stages, SSCGAN effectively captures this hierarchical nature of facial attributes.

Results and Performance

Evaluated on the CelebA dataset, SSCGAN demonstrates superior performance over established models like StarGAN, AttGAN, and STGAN.

  • Accuracy: SSCGAN achieved a classification accuracy of 94.21%, a substantial improvement over STGAN (83.21%) and RelGAN (83.59%).
  • Fidelity: In terms of image quality, SSCGAN recorded a Frechet Inception Distance (FID) of 4.69, outperforming the previous state-of-the-art.
  • Qualitative Quality: Visual inspections show that SSCGAN produces cleaner edits. When adding bangs, for example, the model avoids common artifacts like forehead ghosting or background distortion, which frequently plague models relying on local concatenation.

Conclusion

SSCGAN represents a significant step forward in facial attribute editing by treating attributes as global styles rather than local labels. By balancing global style shifts with local spatial preservation, it achieves high-fidelity results that respect both the target attribute and the original identity. While currently optimized for facial images, the underlying principles of style skip connections hold promise for broader applications in image-to-image translation and semantic segmentation.

Summary updated

Abstract

Existing facial attribute editing methods typically employ an encoder-decoder architecture where the attribute information is expressed as a conditional one-hot vector spatially concatenated with the image or intermediate feature maps. However, such operations only learn the local semantic mapping but ignore global facial statistics. In this work, we focus on solving this issue by editing the channel-wise global information denoted as the style feature. We develop a style skip connection based generative adversarial network, referred to as SSCGAN which enables accurate facial attribute manipulation. Specifically, we inject the target attribute information into multiple style skip connection paths between the encoder and decoder. Each connection extracts the style feature of the latent feature maps in the encoder and then performs a residual learning based mapping function in the global information space guided by the target attributes. In the following, the adjusted style feature will be utilized as the conditional information for instance normalization to transform the corresponding latent feature maps in the decoder. In addition, to avoid the vanishing of spatial details (e.g. hairstyle or pupil locations), we further introduce the skip connection based spatial information transfer module. Through the global-wise style and local-wise spatial information manipulation, the proposed method can produce better results in terms of attribute generation accuracy and image quality. Experimental results demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.
BibTeX
@inproceedings{chu2020sscgan,
  author = {Wenqing Chu and Ying Tai and Chengjie Wang and Jilin Li and Feiyue Huang and Rongrong Ji},
  title = {{SSCGAN: Facial Attribute Editing via Style Skip Connections}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58555-6\_25},
}
Figure