Abstract
Personalized Talking Face Generation (TFG) often faces a trade-off between the high fidelity of identity-dependent models and the efficiency of identity-agnostic ones. MimicTalk bridges this gap by introducing a framework that creates high-fidelity, expressive 3D avatars in just 15 minutes—a 47x speedup over previous state-of-the-art methods. By adapting a person-agnostic base model through a static-dynamic-hybrid pipeline and employing a novel in-context audio-to-motion model, MimicTalk achieves superior identity similarity and stylistic nuance with minimal training data.
Introduction
The challenge in realistic talking face generation lies in balancing detail and efficiency. Identity-agnostic models offer broad applicability but lack the fine-grained detail of a specific person. Conversely, identity-dependent models like RAD-NeRF produce high-quality results by memorizing individual quirks but require hours of training and extensive high-quality footage. Furthermore, these models often struggle with out-of-domain inputs, such as cross-lingual lip-syncing. MimicTalk aims to provide the deep personalization of person-specific models with the efficiency and robustness of generic architectures.
Methodology
MimicTalk's efficiency stems from two primary innovations:
- SD-Hybrid Identity Adaptation: Rather than training a Neural Radiance Field (NeRF) from scratch, MimicTalk adapts a pre-trained, person-agnostic 3D model (Real3D-Portrait). This adaptation is split into two phases: Static Adaptation (Tri-Plane Inversion) to capture unique static features like wrinkles and hair, and Dynamic Adaptation (LoRA) to learn person-specific facial muscle movements during a brief 15-minute fine-tuning period.
- In-Context Stylized Audio-to-Motion (ICS-A2M): To capture unique talking styles without explicit style vectors, MimicTalk uses Flow Matching and an in-context learning approach. By using a short reference clip as a prompt, the model mimics the style found in the reference to generate motion for new audio.
Experiments and Results
The model was trained on the VoxCeleb2 dataset for general motion and evaluated on personalized benchmarks, including high-quality 3-minute videos. MimicTalk demonstrates a significant leap in performance. Adaptation to a new identity requires only 15 minutes on a single NVIDIA A100, compared to approximately 13 hours for RAD-NeRF. Despite this speedup, it achieves higher identity similarity (CSIM) and superior synchronization scores (SyncNet) than existing baselines.
Key Insights
- Residual Learning: By starting with a person-agnostic base, the model only needs to learn the "residuals" that make an individual unique, rather than relearning general facial structures.
- In-Context Nuance: Treating style as a motion in-filling task preserves more nuance than compressing style into a single vector.
- Generative Motion: Flow Matching allows for more expressive and diverse facial motions compared to deterministic mappings.
Conclusion and Future Work
MimicTalk successfully reduces the barrier to creating personalized 3D talking faces. While highly effective for facial expressions, current limitations include naive modeling of hair and torso movements and sub-real-time inference speeds (15 FPS). Future work will explore Gaussian Splatting for real-time performance and expanded control over non-facial gestures.
Summary updated