M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation

Kui Jiang, Shiyu Liu, Junjun Jiang, Xin Yang, Hongxun Yao, Xiaopeng FanarXiv 2025 · DOI · arXiv

Abstract

Creating realistic talking heads from audio is a significant challenge in digital human synthesis. Existing methods often suffer from temporal jitter, kinematic discontinuity between the head and torso, and rendering artifacts like color leakage. M2DAO-Talker addresses these issues by leveraging 3D Gaussian Splatting (3DGS) within a framework that decouples multi-granular motions and employs an alternating optimization strategy. This approach achieves high-fidelity, real-time results (150 FPS) while eliminating common artifacts.

Problem Statement

While Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have advanced talking head generation, three primary obstacles remain:

  1. Temporal Jitter: Pose estimation is often entangled with facial expressions, leading to shaky head movements.
  2. Kinematic Discontinuity: The head and torso frequently appear disconnected, causing "piercing" artifacts where the neck clips through clothing.
  3. Rendering Artifacts: Joint training of the mouth and face can lead to "color leakage" (e.g., lip color bleeding into teeth) or blurred edges due to imprecise segmentation.

Proposed Methodology

The M2DAO-Talker framework reformulates the generation task into a unified three-step process: Video Preprocessing, Motion Representation, and Rendering Reconstruction.

Overview of the M2DAO-Talker framework, illustrating the three main stages: Video Preprocessing, Motion Representation, and Rendering Reconstruction.

Multi-granular Motion Decoupling

Instead of treating the head as a single entity, the model separates motion into three distinct granularities: rigid head rotation, non-rigid facial expressions (Face Branch), and fine-grained oral articulations (Inside Mouth Branch). To ensure stability, the system uses Laplacian-filtered optical flow to identify deformation-invariant regions (like ears and hairlines) for accurate head pose estimation (HPE).

Optimization and Consistency

  • Motion Consistency Constraint (MCC): This aligns facial deformations with global torso dynamics. By optimizing a transparency parameter ($\alpha$) for each Gaussian, the model creates smooth alpha mattes that blend the face and torso naturally, preventing the "floating head" effect.
  • Alternating Optimization Strategy (AOS): To prevent imbalanced supervision that causes lip discoloration, the model cyclically updates the Face and Inside Mouth branches. This allows the network to refine overlapping regions without destructive interference.

Key Insights

  • Flow-based Stability: Using optical flow on rigid areas (ears/hair) provides a cleaner signal for head rotation than traditional landmark-based methods, which drift during speech.
  • Dedicated Oral Branch: Treating the "Inside Mouth" as a dedicated branch with a teeth-aware parser results in significantly sharper dental and lingual rendering.
  • Biomechanical Coherence: Explicitly coupling facial motion with torso dynamics through the MCC ensures a physically plausible digital human.

Experimental Results

M2DAO-Talker sets a new benchmark across several quantitative and qualitative metrics:

  • Visual Quality: Achieved a 34.47 dB PSNR, a 2.43 dB improvement over the previous state-of-the-art, TalkingGaussian.
  • Sync Quality: It achieved a 7.756 Sync-C score, outperforming specialized 2D lip-sync models.
  • Efficiency: The model runs at 150 FPS during inference, making it suitable for real-time applications.
Qualitative comparison of M2DAO-Talker against state-of-the-art methods, highlighting improvements in lip synchronization and torso-head integration.

Qualitative evaluations demonstrate that M2DAO-Talker effectively eliminates the "piercing" artifacts seen in competing models (indicated by yellow arrows in Figure 5) and provides superior lip synchronization (indicated by dashed boxes).

Limitations and Future Work

Despite its performance, the model faces challenges with audio robustness; noisy input can occasionally cause lip jitter. Additionally, the current use of binary semantic masks can lead to jagged edges, suggesting a need for "alpha-aware" masks in the future. While currently identity-specific, future research aims to extend this framework to "one-shot" generation for arbitrary avatars.

Summary updated

Abstract

Audio-driven talking head generation holds significant potential for film production. While existing 3D methods have advanced motion modeling and content synthesis, they often produce rendering artifacts, such as motion blur, temporal jitter, and local penetration, due to limitations in representing stable, fine-grained motion fields. Through systematic analysis, we reformulate talking head generation into a unified framework comprising three steps: video preprocessing, motion representation, and rendering reconstruction. This framework underpins our proposed M2DAO-Talker, which addresses current limitations via multi-granular motion decoupling and alternating optimization. Specifically, we devise a novel 2D portrait preprocessing pipeline to extract frame-wise deformation control conditions (motion region segmentation masks, and camera parameters) to facilitate motion representation. To ameliorate motion modeling, we elaborate a multi-granular motion decoupling strategy, which independently models non-rigid (oral and facial) and rigid (head) motions for improved reconstruction accuracy. Meanwhile, a motion consistency constraint is developed to ensure head-torso kinematic consistency, thereby mitigating penetration artifacts caused by motion aliasing. In addition, an alternating optimization strategy is designed to iteratively refine facial and oral motion parameters, enabling more realistic video generation. Experiments across multiple datasets show that M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness versus TalkingGaussian while with 150 FPS inference speed. Our project homepage is https://m2dao-talker.github.io/M2DAO-Talk.github.io.
BibTeX
@misc{jiang2025m,
  author = {Kui Jiang and Shiyu Liu and Junjun Jiang and Xin Yang and Hongxun Yao and Xiaopeng Fan},
  title = {{M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation}},
  year = {2025},
  eprint = {2507.08307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.08307},
}
Figure