Abstract
Neural face reenactment aims to transfer the head pose and facial expressions of a target image to a source image while faithfully preserving the source's identity. This paper introduces StyleMask, a novel framework that operates within the highly disentangled "Style Space" ($S$) of a pre-trained StyleGAN2. By utilizing a specialized Mask Network and 3D Morphable Model (3DMM) supervision, StyleMask eliminates the need for paired training data and achieves state-of-the-art results in identity preservation and image quality, even under extreme pose variations.
Introduction
Neural face reenactment is the task of "driving" a source face with the movements and expressions of a target face. The fundamental challenge in this domain is disentanglement: the ability to separate "identity" (facial shape, skin texture, and hair) from "pose" (head orientation and muscle movements).
Existing methods often struggle with three primary issues:
- Data Dependency: Many models require "paired" training data—videos of the same individual performing various movements—which is difficult to collect at scale.
- Identity Leakage: When source and target identities differ significantly, the resulting image often becomes a hybrid of both individuals.
- Extreme Poses: Many models fail or produce significant artifacts when the target head turns at sharp angles.
StyleMask addresses these challenges by leveraging the generative power of StyleGAN2, specifically by manipulating its latent representations rather than training a generator from scratch.
Methodology
The Style Space ($S$)
While most StyleGAN research utilizes the $W$ or $W+$ latent spaces, StyleMask operates in the Style Space ($S$). Recent findings suggest that $S$ is significantly more disentangled than its predecessors; changing a single channel in $S$ often corresponds to a specific, isolated semantic change in the generated image. The authors demonstrate that $S$ provides the granularity required for high-fidelity reenactment that $W$ lacks.
The Mask Network
The core of the StyleMask architecture is a Mask Network. Given the style codes of a source ($s_s$) and a target ($s_t$), the network learns to produce a mask ($m$) that acts as a channel-wise selector:
$$s_r = m \odot s_t + (1 - m) \odot s_s$$
The resulting reenacted code ($s_r$) inherits pose-related channels from the target and identity-related channels from the source. To account for the hierarchical nature of StyleGAN2, the authors employ a multi-head Mask Network where different sub-networks handle different layers (e.g., coarse layers for global pose and fine layers for texture).
Supervision and Consistency
To train the Mask Network without paired data, the authors utilize a 3D Morphable Model (3DMM). They generate random synthetic images, extract their pose and identity coefficients via 3DMM, and create a "ground truth" 3D shape that combines the source's identity with the target's pose. The loss function then encourages the reenacted image to match this synthesized 3D geometry.
Furthermore, to mitigate artifacts during large pose changes, a recurrent cycle consistency loss is introduced. By reenacting the source with a target and then using that result as a new source to be reenacted again, the model ensures that identity characteristics remain stable across multiple transformations.
Experiments and Results
Datasets
The model was pre-trained on FFHQ and the Mask Network was trained entirely on synthetic images generated by the GAN. Evaluation was conducted on CelebA-HQ for real-world performance and VoxCeleb1 & 2 for comparison against video-based baselines.
Performance Metrics
StyleMask outperformed several state-of-the-art methods, including Fast Bi-layer, PIR, and StyleFusion, across multiple metrics:
- Identity Preservation (CSIM): Achieved a score of 0.70, indicating superior retention of the source's identity.
- Image Quality (FID): Achieved a score of 10.0, significantly outperforming competitors like PIR (27.4) and Fast Bi-layer (90.0).
- User Preference: In a user study, StyleMask was preferred 46% of the time, nearly double the score of the next closest method (26%).
Qualitatively, StyleMask excels at preserving source-specific details such as eyeglasses and hair accessories, even when these elements are absent in the target image.
Conclusion and Limitations
StyleMask demonstrates that the Style Space of StyleGAN2 is a powerful tool for disentangled face reenactment. By training on synthetic data with 3DMM supervision, the framework avoids the pitfalls of paired data collection. However, the model is bounded by the capabilities of the underlying pre-trained generator. It may struggle with "extreme" expressions not present in the FFHQ training distribution. Additionally, the quality of reenactment for real photos is dependent on the accuracy of the initial GAN inversion process.
Summary updated