Abstract
The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image, which may be of a different person (cross-reenactment). Most existing methods are CNN-based and estimate optical flow from the source image to the current driving frame, which is then inpainted and refined to produce the output animation. We propose a transformer-based encoder for computing a set-latent representation of the source image(s). We then predict the output color of a query pixel using a transformer-based decoder, which is conditioned with keypoints and a facial expression vector extracted from the driving frame. Latent representations of the source person are learned in a self-supervised manner that factorize their appearance, head pose, and facial expressions. Thus, they are perfectly suited for cross-reenactment. In contrast to most related work, our method naturally extends to multiple source images and can thus adapt to person-specific facial dynamics. We also propose data augmentation and regularization schemes that are necessary to prevent overfitting and support generalizability of the learned representations. We evaluated our approach in a randomized user study. The results indicate superior performance compared to the state-of-the-art in terms of motion transfer quality and temporal consistency.
Introduction
Face reenactment—the process of transferring the motion and expressions of a "driving" person to a "source" person—is a cornerstone of digital puppetry and low-bandwidth video conferencing. However, the field has long been plagued by two major issues:
- Identity Leakage: In "cross-reenactment" (where the source and driving persons are different), the synthesized output often starts to look like a hybrid of both people. The driving person’s face shape "leaks" into the source person’s identity.
- Warping Artifacts: Most existing methods use CNNs to estimate optical flow and "warp" pixels from the source to the target. This often fails during large head rotations or complex expressions, leading to blurry patches or distorted features.
The authors of FSRT aim to solve these by moving away from pixel-warping and toward a transformer-based representation that understands the face as a set of factorized features.
Methods
FSRT (Facial Scene Representation Transformer) adapts the Scene Representation Transformer (SRT) architecture for the specific nuances of human faces. The pipeline treats face reenactment as a neural rendering task rather than a traditional image-warping problem.
The architecture consists of three main stages:
- The Encoder: It processes one or more source images along with their extracted keypoints and expression vectors. Instead of a single feature map, it produces a set-latent representation—a collection of vectors that describe the "scene" of the face. Because it uses a transformer, the model can naturally accept an arbitrary number of source images, allowing it to "see" the source person from multiple angles.
- The Decoder: To generate the output, the decoder queries every pixel location. It is conditioned on the driving frame’s head pose (keypoints) and expression vector. It uses cross-attention to "look" at the set-latent representation and decide what color each pixel should be.
- Factorization & Regularization: To prevent identity leakage, the authors introduce Statistical Regularization (inspired by VICReg). This forces the model to decorrelate expression features from identity features. They also use heavy data augmentation (color jittering and random cropping) to ensure the model doesn't rely on background clues or specific lighting from the driving video.
Datasets
The model was trained and evaluated on several standard benchmarks to ensure robustness and generalizability:
- VoxCeleb & VoxCeleb2: Large-scale video datasets of celebrities speaking, used for primary training and testing.
- CelebA-HQ: Used to provide high-quality static source images for testing generalization.
- CelebV: A more recent video dataset used to demonstrate the model's ability to generalize to diverse identities and environments.
Key Insights
- Faces as Sets, Not Grids: By treating the face as a set of latent vectors rather than a fixed 2D grid, the model gains the flexibility to handle occlusions and large pose changes much better than flow-based CNNs.
- The Power of Multi-Source: Most models are "one-shot" (one source image). FSRT is "n-shot" by design. If provided with multiple photos of a person—front, left, and right—the transformer encoder fuses that information into a richer latent representation, leading to much more accurate profile views.
- Decoupling is Essential: The authors realized that the expression vector should not contain information about the person's face shape. Their statistical regularization loss specifically penalizes the model if the expression vector starts carrying identity-specific information.
Key Results
FSRT demonstrates significant improvements over previous state-of-the-art (SOTA) models like FOMM, DaGAN, and TSMM:
- Quantitative Superiority: On the VoxCeleb test set, FSRT achieved a lower Average Keypoint Distance (AKD) and higher SSIM/PSNR than most competitors. Specifically, its multi-source version (2-Src) reached a PSNR of 25.00, significantly outperforming the single-source SOTA.
- User Preference: In a randomized user study, participants preferred FSRT's results 87% to 99% of the time compared to other methods, citing better identity preservation and more natural-looking expressions.
- Efficiency: Despite being a transformer, a "small decoder" version of the model can run at 23 FPS on a single NVIDIA RTX 4090, making it viable for real-time applications.
Limitations and Future Work
While FSRT produces remarkably sharp faces, it still faces some hurdles. If the driving person performs an extreme expression not present in the training data, the model struggles to reconstruct it accurately. Furthermore, very fine details like individual hair strands can sometimes be slightly softer than in models that directly warp source pixels. The authors suggest that integrating 3D geometry (like volume rendering or NeRF-style approaches) could further improve the consistency of fine details and background reconstruction.
Summary updated