Abstract
In recent advances of deep generative models, face reenactment—manipulating and controlling human face, including their head movement—has drawn much attention for its wide range of applicability. Despite its strong expressiveness, it is inevitable that the models fail to reconstruct or accurately generate unseen side of the face of a given single reference image. Most of existing methods alleviate this problem by learning appearances of human faces from large amount of data and generate realistic texture at inference time. Rather than completely relying on what generative models learn, we show that simple extension by using multiple reference images significantly improves generation quality. We show this by 1) conducting the reconstruction task on publicly available dataset, 2) conducting facial motion transfer on our original dataset which consists of multi-person's head movement video sequences, and 3) using a newly proposed evaluation metric to validate that our method achieves better quantitative results.
Introduction
Face reenactment, or facial motion transfer, involves animating a "reference" person's face using the movements and expressions of a "driving" video. While purely 2D-based generative models have become highly effective, they suffer from a fundamental geometric limitation: if the driving video reveals a side of the face not present in a single reference image, the model must "hallucinate" the missing textures. This often leads to blurry or distorted artifacts.
Furthermore, standard metrics like the Fréchet Inception Distance (FID) only measure the overall distribution of image quality and do not verify if the generated face accurately matches the driving video's pose. To address these issues, this paper proposes using multiple reference images and a novel evaluation framework.
Methodology
The authors build their architecture on top of the First Order Motion Model (FOMM). Instead of a single reference, the system accepts $K$ reference images. A Fusion Unit is introduced to combine the warped feature maps of these multiple references before they are processed by the decoder.
Fusion Strategies
Two attention-based fusion strategies were investigated to intelligently blend reference features:
- Patch-wise weighted sum: The network generates a single-channel spatial mask for each reference image, acting like a 2D alpha-blend to choose which spatial regions (patches) to pull from which reference.
- Element-wise weighted sum: A more granular approach where the mask has multiple channels matching the depth of the feature maps. This allows the network to assign different blending weights to every single pixel and every single feature channel, pulling specific textural details from the most relevant reference image.
Evaluation and Datasets
The model was evaluated using VoxCeleb1 for reconstruction tasks and a custom Internal Dataset for motion transfer. The internal dataset included precise hardware-recorded head angles (yaw, pitch, roll) to facilitate a new evaluation method.
Bin-based Angle-wise Evaluation
To fix the flaws in standard generative metrics, the authors introduced Bin-based angle-wise evaluation. Generated and ground-truth frames are grouped into narrow angular "bins." Perceptual similarity (LPIPS) is then calculated strictly between images within the same pose bin, ensuring the metric reflects actual pose-matching accuracy rather than just general image quality.
Key Results
Quantitative results show that providing multiple references and using granular fusion leads to significantly more faithful results:
- Reconstruction (VoxCeleb1): Using 3 reference images, the element-wise fusion model achieved an Average Keypoint Distance (AKD) of 1.156 and an Average Euclidean Distance (AED) of 0.078, significantly outperforming the baseline FOMM (AKD: 1.294, AED: 0.140).
- Motion Transfer (Internal Data): Evaluated using the bin-based LPIPS metric (lower is better), the element-wise model outperformed the baseline across all movement axes: Yaw (0.085 vs 0.117), Pitch (0.107 vs 0.131), and Roll (0.141 vs 0.160).
Conclusion
By shifting the burden from generative hallucination to feature-level fusion of multiple references, this method achieves higher fidelity in face reenactment. The element-wise strategy proves superior by allowing the network to mix and match references on a per-channel basis. Future work will explore making the system flexible to an arbitrary number of reference images, allowing quality to scale with available data.
Summary updated