Abstract
Creating realistic talking heads from audio is a significant challenge in digital human synthesis. Existing methods often suffer from temporal jitter, kinematic discontinuity between the head and torso, and rendering artifacts like color leakage. M2DAO-Talker addresses these issues by leveraging 3D Gaussian Splatting (3DGS) within a framework that decouples multi-granular motions and employs an alternating optimization strategy. This approach achieves high-fidelity, real-time results (150 FPS) while eliminating common artifacts.
Problem Statement
While Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have advanced talking head generation, three primary obstacles remain:
- Temporal Jitter: Pose estimation is often entangled with facial expressions, leading to shaky head movements.
- Kinematic Discontinuity: The head and torso frequently appear disconnected, causing "piercing" artifacts where the neck clips through clothing.
- Rendering Artifacts: Joint training of the mouth and face can lead to "color leakage" (e.g., lip color bleeding into teeth) or blurred edges due to imprecise segmentation.
Proposed Methodology
The M2DAO-Talker framework reformulates the generation task into a unified three-step process: Video Preprocessing, Motion Representation, and Rendering Reconstruction.
Multi-granular Motion Decoupling
Instead of treating the head as a single entity, the model separates motion into three distinct granularities: rigid head rotation, non-rigid facial expressions (Face Branch), and fine-grained oral articulations (Inside Mouth Branch). To ensure stability, the system uses Laplacian-filtered optical flow to identify deformation-invariant regions (like ears and hairlines) for accurate head pose estimation (HPE).
Optimization and Consistency
- Motion Consistency Constraint (MCC): This aligns facial deformations with global torso dynamics. By optimizing a transparency parameter ($\alpha$) for each Gaussian, the model creates smooth alpha mattes that blend the face and torso naturally, preventing the "floating head" effect.
- Alternating Optimization Strategy (AOS): To prevent imbalanced supervision that causes lip discoloration, the model cyclically updates the Face and Inside Mouth branches. This allows the network to refine overlapping regions without destructive interference.
Key Insights
- Flow-based Stability: Using optical flow on rigid areas (ears/hair) provides a cleaner signal for head rotation than traditional landmark-based methods, which drift during speech.
- Dedicated Oral Branch: Treating the "Inside Mouth" as a dedicated branch with a teeth-aware parser results in significantly sharper dental and lingual rendering.
- Biomechanical Coherence: Explicitly coupling facial motion with torso dynamics through the MCC ensures a physically plausible digital human.
Experimental Results
M2DAO-Talker sets a new benchmark across several quantitative and qualitative metrics:
- Visual Quality: Achieved a 34.47 dB PSNR, a 2.43 dB improvement over the previous state-of-the-art, TalkingGaussian.
- Sync Quality: It achieved a 7.756 Sync-C score, outperforming specialized 2D lip-sync models.
- Efficiency: The model runs at 150 FPS during inference, making it suitable for real-time applications.
Qualitative evaluations demonstrate that M2DAO-Talker effectively eliminates the "piercing" artifacts seen in competing models (indicated by yellow arrows in Figure 5) and provides superior lip synchronization (indicated by dashed boxes).
Limitations and Future Work
Despite its performance, the model faces challenges with audio robustness; noisy input can occasionally cause lip jitter. Additionally, the current use of binary semantic masks can lead to jagged edges, suggesting a need for "alpha-aware" masks in the future. While currently identity-specific, future research aims to extend this framework to "one-shot" generation for arbitrary avatars.
Summary updated