Introduction
In the field of digital human synthesis, "talking head" generation has traditionally focused on lip-syncing and facial expressions. However, realistic human communication is a holistic process involving shoulder shrugs, torso shifts, and expressive hand gestures. Existing 2D speaking avatar frameworks often struggle to maintain identity consistency and structural integrity when moving beyond the face. Common issues include "robotic" torso movements, textural flickering where person-specific details (like suit patterns) drift during motion, and the notorious "spaghetti finger" problem—where generative models fail to render anatomically correct hands during complex movements.
To address these challenges, we introduce TALK-Act (Motion-Enhanced Textural-Aware ModeLing for SpeaKing Avatar Reenactment). TALK-Act is a diffusion-based framework designed to generate high-fidelity, temporally stable videos of speaking avatars by controlling the entire upper body. By utilizing a novel textural alignment module and a specialized hand-memory bank, the system achieves state-of-the-art performance using as little as 30 seconds of person-specific footage.
Methodology
TALK-Act utilizes a two-branch diffusion architecture consisting of a Denoising Branch for video generation and a Reference Branch to preserve the subject's identity. The framework introduces three primary innovations to ensure structural and textural fidelity.
Hybrid Structural Guidance
Rather than relying on simplistic 2D skeletons, TALK-Act employs a dense geometric map. This hybrid guidance combines 2D poses for the body with 3D facial meshes and 3D hand meshes. This multi-layered approach provides the diffusion model with a rigorous spatial framework, ensuring that movements remain anatomically plausible.
Motion-Enhanced Textural Alignment (META)
The core innovation of TALK-Act is the META module. Traditional models often struggle to map static textures from a reference image onto a moving target. META solves this by building a correspondence matrix between the reference frame and the driving motion. By aligning the motion of the reference person to the motion of the driving video, the model creates a "motion-to-motion" bridge. This allows the system to accurately "paste" textures from the reference image onto the corresponding parts of the moving avatar, significantly reducing identity drift and flickering.
Memory-based Hand Recovering
To resolve the common failure point of hand synthesis, TALK-Act incorporates a learnable global memory bank of hand textures. During the generation process, the model queries this bank to retrieve high-quality, pose-specific hand details. This ensures that even during rapid or complex gestures, the hands remain clear and anatomically correct.
Experimental Setup
The researchers evaluated TALK-Act using a curated dataset of 11 subjects. The model was pre-trained on approximately 28 hours of footage from five identities sourced from the PATS dataset. For evaluation, six additional individuals were used for few-shot adaptation tests, including both PATS identities and self-recorded videos. All data was processed at 512x512 resolution at 25 FPS, with landmarks extracted via DWPose.
Results and Discussion
TALK-Act demonstrates significant improvements over existing state-of-the-art (SOTA) models such as AnimateAnyone and Make-Your-Anchor.
Quantitative Performance
- Visual Quality: TALK-Act achieved a Fréchet Inception Distance (FID) of 18.87, a substantial improvement over AnimateAnyone's 29.66.
- Temporal Stability: In terms of Fréchet Video Distance (FVD), TALK-Act scored 145.40, vastly outperforming the 416.90 recorded by previous top-tier models.
- Structural Accuracy: Landmark distance metrics for both faces and hands were significantly lower, indicating that the avatar follows the driving motion with high precision.
Qualitative Insights
Qualitatively, the model excels at preserving complex clothing textures and maintaining facial identity during rapid hand movements—scenarios where baseline models typically fail. The use of Biased Attention within the META module allows the transformer-based mechanism to focus on specific regions (e.g., focusing on the reference hand when drawing the target hand), leading to much sharper details.
Limitations and Future Work
While TALK-Act represents a major step forward, certain limitations remain. The model is sensitive to unstable camera movements and complex background changes in the driving video. Additionally, interactions with physical objects (e.g., a person holding a phone) are not yet handled consistently, often resulting in poor hand-object contact rendering. Future research will focus on disentangling body movements from background dynamics to improve robustness for "in-the-wild" video applications.
Conclusion
TALK-Act provides a robust solution for 2D speaking avatar reenactment by prioritizing textural awareness and structural guidance. By bridging the gap between motion and texture through the META module and addressing hand synthesis via a dedicated memory bank, it enables the creation of lifelike digital avatars with minimal source data, paving the way for more immersive and accessible digital communication.
Summary updated