Single Source One Shot Reenactment using Weighted Motion from Paired Feature Points

Soumya Tripathy, Juho Kannala, Esa Rahtu

TL;DR

This paper introduces a novel "one-shot" image reenactment model that allows a single source image to mimic the motion of a driver video without losing the source's unique identity. By using a Transformer-based paired feature point detector and a weighted motion model (Moving Least Squares), the system achieves superior identity preservation and robustness compared to previous state-of-the-art methods.

What Problem Does This Paper Address?

Image reenactment—making a "source" image (like a portrait) follow the movements of a "driver" video—is a staple of modern AI, used in everything from movie production to digital avatars. However, two major hurdles remain:

  1. Identity Leakage: Most models struggle with "cross-person" reenactment. If the driver has a vastly different face shape than the source, the resulting animation often looks like a "hybrid" of the two, losing the source's original structure.
  2. Keypoint Sensitivity: Current models often rely on unsupervised keypoints that track facial contours. If a single keypoint is predicted incorrectly due to noise or occlusion, the local image area becomes severely distorted.

The authors aim to decouple motion from identity so that the source image moves like the driver but stays looking like the source.

Methods

The authors propose a pipeline consisting of five main steps, featuring two significant innovations:

  • Paired-Feature-Point Estimation: Unlike standard models that find keypoints on the source and driver images independently, this model uses a Transformer module with a co-attention layer. It looks at both images simultaneously to predict "anchors" that describe the motion between them. This makes the points "motion-centric" rather than "shape-centric," preventing the driver's facial geometry from leaking into the source.
  • Weighted Motion Model (Moving Least Squares): In traditional models (like the First Order Motion Model), a pixel’s movement is often dictated by the single nearest keypoint. Here, the authors use Moving Least Squares (MLS). Every pixel’s motion is calculated as a weighted sum of all paired feature points. This global approach ensures that if one point is slightly off, the influence of the other points "smooths out" the error.
  • The Pipeline:
    1. Image Embedder: Maps images to a latent space.
  • Transformer: Predicts the paired feature points.
  • Motion Estimation: Uses MLS to define the transformation.
  • Warping Field Generator: Creates a dense warping field and an occlusion mask.
  • Generation Module: An occlusion-aware generator produces the final reenacted image.

Datasets

The model was trained and evaluated on several diverse datasets to prove it is object-agnostic:

  • VoxCeleb & FaceForensics++: For facial reenactment and cross-person identity testing.
  • MGif: A dataset of animal animations (e.g., running horses) to test non-face objects.
  • Tai-chi-HD: Full-body human motion to test complex, non-rigid deformations.

Key Insights and Ideas

  • Joint Prediction is Key: By forcing the model to look at the source and driver together via attention, the feature points become relative markers of change rather than absolute markers of anatomy.
  • Redundancy Equals Robustness: By moving from a "local-only" influence model to a "global-weighted" one, the system becomes much more tolerant of noisy keypoint predictions.
  • Shape Independence: Because the feature points don't strictly follow contours (like lip corners or chin lines), they carry less "identity information," which is exactly what you want when you're trying to transfer motion without transferring the face shape.

Key Results

  • Identity Preservation: In quantitative tests using Identity Cosine Similarity (ISIM), the proposed model scored 0.70, significantly outperforming the popular First Order Motion Model (FOMM) at 0.57.
  • Superior Robustness: To test stability, the researchers added random noise to the feature points. FOMM’s error rate spiked by 97.3%, while the proposed model’s error only increased by 22.9%, proving the weighted motion model is far more stable.
  • Qualitative Excellence: Visual results show that the model successfully animates faces, horses, and humans while keeping the source's original proportions intact, even when the driver's proportions are very different.

Limitations and Future Work

While the model excels at identity preservation, the authors note that some existing models might still achieve slightly higher scores in Pose Similarity (PSIM). This is a trade-off: models that "leak" more of the driver's identity often follow the driver's exact landmarks more closely.

Future work could involve refining the balance between perfect pose replication and identity preservation, as well as exploring even more complex scenarios involving extreme head poses or high-frequency textures that are currently difficult for one-shot generators to reconstruct perfectly.

Summary updated

BibTeX
@inproceedings{tripathy2022single,
  author = {Soumya Tripathy and Juho Kannala and Esa Rahtu},
  title = {{Single Source One Shot Reenactment using Weighted Motion from Paired Feature Points}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00218},
}
Figure