One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space

Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, Georgios Tzimiropoulos

Abstract

Neural face reenactment aims to transfer the pose and expression of a target person to a source image while preserving the source's identity. Traditional methods often struggle with identity leakage or lack photorealism. This paper presents a novel framework that leverages the high-quality latent space of a pre-trained StyleGAN2. By discovering controllable directions corresponding to 3D facial parameters, the proposed method achieves state-of-the-art performance in identity preservation and reenactment quality. A joint training scheme and feature space refinement further enhance the model's ability to handle real-world images and complex backgrounds in near real-time.

Introduction

Neural face reenactment is the task of animating a source image of a person using the head pose and facial expressions of a target "driving" video. A primary challenge in this field is disentanglement: modifying the target's motion without altering the source's identity or fine details. While previous approaches have attempted to learn these disentangled representations from scratch, they often fall short in photorealism or identity stability. This work proposes to leverage the rich, pre-trained latent space of StyleGAN2, identifying specific "controllable directions" that correspond to 3D facial parameters.

Methodology

The proposed framework avoids training a generator from scratch, instead utilizing a pre-trained StyleGAN2 model. The core innovation lies in the discovery of latent directions that map to physical facial changes.

Latent Direction Discovery and 3D Guidance

To achieve precise control, the authors use a 3D Morphable Model (Net3D) to extract pose and expression parameters ($\mathbf{p}_s, \mathbf{p}_t$). A transformation matrix $A$ is learned to map the difference in these 3D parameters ($\Delta \mathbf{p}$) to a shift in the GAN's latent space ($\Delta \mathbf{w}$). Because 3D models are inherently disentangled, they provide a robust geometric prior for steering the GAN.

The full reenactment pipeline, featuring the Latent Edit Module which computes latent shifts based on 3D facial parameters.

Joint Training and Feature Refinement

For real-world application, the system incorporates a GAN inversion encoder ($E_w$) to map input images into the latent space. A "Joint Training" scheme optimizes the encoder and the transformation matrix $A$ simultaneously, removing the need for time-consuming per-image optimization during inference. To preserve high-frequency details like hair textures and backgrounds—which are often lost in standard latent codes—a Feature Transformation (FT) module refines the generator's internal feature maps.

Experimental Results

The model was evaluated on the VoxCeleb1, VoxCeleb2, FaceForensics, and 300-VW datasets. Specialized benchmarks (Benchmark-L and XL) were used to test robustness in extreme pose scenarios.

Qualitative and Quantitative Performance

The proposed method significantly outperforms existing state-of-the-art models. In user studies, the method was preferred by 52.1% of participants, far exceeding competitors. Quantitatively, it achieved top scores in identity preservation (CSIM) and pose accuracy (ARD/AED).

Qualitative comparison showing the improvements gained from Joint Training and Feature Space Refinement (FSR) over standard optimization methods.

Efficiency

The "Joint Training" approach enables "optimization-free" inference, allowing the system to generate a reenacted frame in approximately 0.13 seconds. This efficiency makes the framework suitable for near-real-time applications.

Conclusion

By finding linear directions in a pre-trained GAN's latent space guided by 3D geometry, this framework achieves high-fidelity face reenactment with exceptional identity preservation. While challenges remain regarding complex accessories and dataset biases, the method sets a new standard for one-shot neural face animation.

Summary updated

BibTeX
@article{bounareli2024one,
  author = {Stella Bounareli and Christos Tzelepis and Vasileios Argyriou and Ioannis Patras and Georgios Tzimiropoulos},
  title = {{One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space}},
  journal = {Int. J. Comput. Vis.},
  year = {2024},
  doi = {10.1007/s11263-024-02018-6},
}
Figure