StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment

Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, Georgios TzimiropoulosZenodo 2022 · DOI · OpenAlexGitHub resource

Abstract

Neural face reenactment aims to transfer the head pose and facial expressions of a target image to a source image while faithfully preserving the source's identity. This paper introduces StyleMask, a novel framework that operates within the highly disentangled "Style Space" ($S$) of a pre-trained StyleGAN2. By utilizing a specialized Mask Network and 3D Morphable Model (3DMM) supervision, StyleMask eliminates the need for paired training data and achieves state-of-the-art results in identity preservation and image quality, even under extreme pose variations.

Introduction

Neural face reenactment is the task of "driving" a source face with the movements and expressions of a target face. The fundamental challenge in this domain is disentanglement: the ability to separate "identity" (facial shape, skin texture, and hair) from "pose" (head orientation and muscle movements).

Existing methods often struggle with three primary issues:

  1. Data Dependency: Many models require "paired" training data—videos of the same individual performing various movements—which is difficult to collect at scale.
  2. Identity Leakage: When source and target identities differ significantly, the resulting image often becomes a hybrid of both individuals.
  3. Extreme Poses: Many models fail or produce significant artifacts when the target head turns at sharp angles.

StyleMask addresses these challenges by leveraging the generative power of StyleGAN2, specifically by manipulating its latent representations rather than training a generator from scratch.

Methodology

The Style Space ($S$)

While most StyleGAN research utilizes the $W$ or $W+$ latent spaces, StyleMask operates in the Style Space ($S$). Recent findings suggest that $S$ is significantly more disentangled than its predecessors; changing a single channel in $S$ often corresponds to a specific, isolated semantic change in the generated image. The authors demonstrate that $S$ provides the granularity required for high-fidelity reenactment that $W$ lacks.

The Mask Network

The core of the StyleMask architecture is a Mask Network. Given the style codes of a source ($s_s$) and a target ($s_t$), the network learns to produce a mask ($m$) that acts as a channel-wise selector:

$$s_r = m \odot s_t + (1 - m) \odot s_s$$

The resulting reenacted code ($s_r$) inherits pose-related channels from the target and identity-related channels from the source. To account for the hierarchical nature of StyleGAN2, the authors employ a multi-head Mask Network where different sub-networks handle different layers (e.g., coarse layers for global pose and fine layers for texture).

Supervision and Consistency

To train the Mask Network without paired data, the authors utilize a 3D Morphable Model (3DMM). They generate random synthetic images, extract their pose and identity coefficients via 3DMM, and create a "ground truth" 3D shape that combines the source's identity with the target's pose. The loss function then encourages the reenacted image to match this synthesized 3D geometry.

Furthermore, to mitigate artifacts during large pose changes, a recurrent cycle consistency loss is introduced. By reenacting the source with a target and then using that result as a new source to be reenacted again, the model ensures that identity characteristics remain stable across multiple transformations.

Experiments and Results

Datasets

The model was pre-trained on FFHQ and the Mask Network was trained entirely on synthetic images generated by the GAN. Evaluation was conducted on CelebA-HQ for real-world performance and VoxCeleb1 & 2 for comparison against video-based baselines.

Performance Metrics

StyleMask outperformed several state-of-the-art methods, including Fast Bi-layer, PIR, and StyleFusion, across multiple metrics:

  • Identity Preservation (CSIM): Achieved a score of 0.70, indicating superior retention of the source's identity.
  • Image Quality (FID): Achieved a score of 10.0, significantly outperforming competitors like PIR (27.4) and Fast Bi-layer (90.0).
  • User Preference: In a user study, StyleMask was preferred 46% of the time, nearly double the score of the next closest method (26%).

Qualitatively, StyleMask excels at preserving source-specific details such as eyeglasses and hair accessories, even when these elements are absent in the target image.

Conclusion and Limitations

StyleMask demonstrates that the Style Space of StyleGAN2 is a powerful tool for disentangled face reenactment. By training on synthetic data with 3DMM supervision, the framework avoids the pitfalls of paired data collection. However, the model is bounded by the capabilities of the underlying pre-trained generator. It may struggle with "extreme" expressions not present in the FFHQ training distribution. Additionally, the quality of reenactment for real photos is dependent on the accuracy of the initial GAN inversion process.

Summary updated

Abstract

In this paper we address the problem of neural face reenactment, where, given a pair of a source and a target facial image, we need to transfer the target's pose (defined as the head pose and its facial expressions) to the source image, by preserving at the same time the source's identity characteristics (e.g., facial shape, hair style, etc), even in the challenging case where the source and the target faces belong to different identities. In doing so, we address some of the limitations of the state-of-the-art works, namely, a) that they depend on paired training data (i.e., source and target faces have the same identity), b) that they rely on labeled data during inference, and c) that they do not preserve identity in large head pose changes. More specifically, we propose a framework that, using unpaired randomly generated facial images, learns to disentangle the identity characteristics of the face from its pose by incorporating the recently introduced style space S of StyleGAN2, a latent representation space that exhibits remarkable disentanglement properties. By capitalizing on this, we learn to successfully mix a pair of source and target style codes using supervision from a 3D model. The resulting latent code, that is subsequently used for reenactment, consists of latent units corresponding to the facial pose of the target only and of units corresponding to the identity of the source only, leading to notable improvement in the reenactment performance compared to recent state-of-the-art methods. In comparison to state of the art, we quantitatively and qualitatively show that the proposed method produces higher quality results even on extreme pose variations. Finally, we report results on real images by first embedding them on the latent space of the pretrained generator.
BibTeX
@inproceedings{bounareli2022stylemask,
  author = {Stella Bounareli and Christos Tzelepis and Vasileios Argyriou and Ioannis Patras and Georgios Tzimiropoulos},
  title = {{StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.8363562},
}

Loading related content...

Figure