Single Source One Shot Reenactment using Weighted motion From Paired Feature Points

Soumya Tripathy, Juho Kannala, Esa Rahtu

Introduction

Image reenactment is the process of animating a static "source" image to follow the movements of a "driver" video. While this technology has advanced significantly, two primary challenges persist: identity leakage and keypoint sensitivity. Identity leakage occurs when the facial structure of the driver "bleeds" into the source image, resulting in a hybrid face that loses the source's unique identity. Keypoint sensitivity refers to the fragility of unsupervised keypoints; if a single point is mispredicted due to noise or occlusion, it can cause severe local distortions in the generated output.

This paper presents a novel one-shot reenactment model designed to decouple motion from identity. By utilizing a Transformer-based paired feature point detector and a global weighted motion model, the system achieves superior identity preservation and robustness compared to existing state-of-the-art methods.

Methodology

The proposed pipeline consists of five integrated modules: an Image Embedder, a Transformer-based point detector, a Motion Estimation module, a Warping Field Generator, and an Occlusion-aware Generation Module.

Paired-Feature-Point Estimation

A core innovation is the shift from independent keypoint detection to Paired-Feature-Point Estimation. Traditional models identify keypoints on the source and driver images separately. In contrast, this model employs a Transformer module with a co-attention layer to analyze both images simultaneously. This joint prediction allows the model to identify "anchors" that describe the relative motion between the two subjects rather than their absolute anatomical shapes. This prevents the driver's facial geometry from leaking into the source's identity.

Paired feature points estimated across different identities to capture motion rather than static geometry.

Weighted Motion Model (Moving Least Squares)

To address keypoint sensitivity, the authors replace local-only influence models with a Weighted Motion Model based on Moving Least Squares (MLS). In previous frameworks, a pixel's movement was often dominated by the single nearest keypoint. The MLS approach calculates each pixel's motion as a weighted sum of all paired feature points. This global perspective ensures that the influence of a noisy or incorrectly predicted point is smoothed out by the collective consensus of the other points, leading to much more stable animations.

Experimental Results

The model was evaluated on diverse datasets, including VoxCeleb and FaceForensics++ for facial reenactment, MGif for animal motion, and Tai-chi-HD for complex human body movements.

Identity Preservation

Quantitative analysis using Identity Cosine Similarity (ISIM) demonstrates the model's effectiveness. The proposed system achieved an ISIM score of 0.70, significantly outperforming the First Order Motion Model (FOMM), which scored 0.57. This indicates that the source's identity is much better preserved during cross-person reenactment.

Robustness to Noise

The stability of the weighted motion model was tested by introducing random noise to the feature points. While FOMM's error rate spiked by 97.3%, the proposed model's error increased by only 22.9%. This four-fold improvement in stability confirms that the global weighted approach is far more resilient to prediction errors.

Discussion and Limitations

While the model excels at identity preservation, there is a known trade-off regarding Pose Similarity (PSIM). Some existing models that allow more identity leakage may follow the driver's landmarks more precisely, resulting in higher PSIM scores. The current approach prioritizes the structural integrity of the source image, which is often more desirable for realistic avatars and digital production. Future research will explore methods to minimize this trade-off, particularly in scenarios involving extreme head poses or complex textures.

Conclusion

By rethinking how motion is captured and applied, this research successfully decouples identity from movement in one-shot image reenactment. The combination of joint feature point prediction and global weighted motion estimation provides a robust framework that maintains the integrity of the source image even under challenging cross-person scenarios.

Summary updated

BibTeX
@inproceedings{tripathy2022single,
  author = {Soumya Tripathy and Juho Kannala and Esa Rahtu},
  title = {{Single Source One Shot Reenactment using Weighted motion From Paired Feature Points}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00218},
}
Figure