One-Shot Pose-Driving Face Animation Platform

He Feng, Donglin Di, Yongjia Ma, Wei Chen, Tonghua SuarXiv 2024 · DOI · OpenReview

Introduction

Creating realistic talking head videos from a single static image—a process known as "one-shot" animation—is a significant challenge in digital human research. Traditional methods often struggle with three primary issues: the need for identity-specific fine-tuning, a lack of expressive nuance in audio-driven movements, and temporal instability that leads to background flickering. This platform addresses these hurdles by integrating specialized spatial and temporal priors into a diffusion-based framework, enabling high-quality video generation without per-identity optimization.

Methodology

The system utilizes a two-stage pipeline designed to decouple facial pose from identity while maintaining high visual fidelity.

Input2Pose Module

This module generates the underlying motion sequence. It supports multi-modal inputs, allowing users to drive animation via an existing video (processed through DWPose) or an audio file (processed via an Audio2Pose module). The output is a facial pose sequence that defines the structural movement of the face and mouth.

Image2Video Module

The rendering engine is built upon a dual U-Net architecture. A Reference Net extracts identity features from the source image, while a Denoising UNet generates the video frames. To ensure stability, the authors introduced a Face Locator to anchor the facial region and a Motion Frame mechanism that trains the network on consecutive frames to reduce jitter.

The overall architecture of the platform, showing the Input2Pose and Image2Video modules.

Implementation and Datasets

The model was trained on two high-resolution datasets: CelebV-HQ, which provides diverse celebrity facial attributes, and HDTF (High-Resolution Talking Face), ensuring the model can handle the complexities of realistic speech and movement. By using a lightweight Pose Guider to map sequences into latent space, the system achieves true one-shot capability, applying complex movements to any face regardless of ethnicity or age.

Key Results

The platform generates talking head videos at a resolution of 512x512 at 24 frames per second. Qualitative evaluations demonstrate that the integration of the Face Locator and Motion Frame mechanism significantly improves temporal consistency and background stability compared to standard baseline methods.

Qualitative results showing the source image, driving signal, and the generated output across various identities.

As shown in the results, the system successfully preserves the identity of the source image while accurately following the pose and expression of the driver, whether the driver is a video or an audio-derived sequence.

Conclusion and Future Work

This platform demonstrates that specific architectural priors can solve common artifacts in diffusion-based video generation without requiring massive retraining. While the current system provides high-quality, expressive results, the computational cost of the dual U-Net architecture remains a bottleneck for real-time applications. Future research will focus on model distillation and architectural streamlining to enable lower-latency inference on consumer-grade hardware.

Summary updated

Abstract

The objective of face animation is to generate dynamic and expressive talking head videos from a single reference face, utilizing driving conditions derived from either video or audio inputs. Current approaches often require fine-tuning for specific identities and frequently fail to produce expressive videos due to the limited effectiveness of Wav2Pose modules. To facilitate the generation of one-shot and more consecutive talking head videos, we refine an existing Image2Video model by integrating a Face Locator and Motion Frame mechanism. We subsequently optimize the model using extensive human face video datasets, significantly enhancing its ability to produce high-quality and expressive talking head videos. Additionally, we develop a demo platform using the Gradio framework, which streamlines the process, enabling users to quickly create customized talking head videos.
BibTeX
@misc{feng2024one,
  author = {He Feng and Donglin Di and Yongjia Ma and Wei Chen and Tonghua Su},
  title = {{One-Shot Pose-Driving Face Animation Platform}},
  year = {2024},
  eprint = {2407.08949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.08949},
}
Figure