TL;DR
OrthoNet is a novel "plug-and-play" framework designed to fix one of the most persistent artifacts in talking face synthesis: distorted, flickering, and unrealistic teeth. By combining a detail-preserving feature extractor with a dual-memory stabilization system, it ensures that teeth remain rigid, clear, and temporally consistent, even during complex lip movements and occlusions.
The Challenge: Crossing the Dental Uncanny Valley
While modern audio-driven Talking Face Synthesis (TFS) has made massive strides in lip-syncing and emotional expression, the "teeth region" remains a major hurdle. Unlike skin or lips, teeth are rigid structures that should not deform. However, current models often treat the mouth as a generic deformable area, leading to two primary issues:
- Temporal Inconsistency: Teeth appear to "grow," "shrink," or flicker between frames because models generate each frame independently or with limited context.
- Generation Hallucinations: Due to frequent occlusion by the lips, models effectively have to "inpaint" the teeth. Without strong priors, this results in blurred textures, missing gaps, or "hallucinated" extra teeth.
Methods: The OrthoNet Architecture
The authors propose OrthoNet, which acts as a virtual "orthodontist" for existing Audio2Video models. It is designed to be added to the end of existing state-of-the-art (SOTA) pipelines (like AniPortrait or EchoMimic), making it highly practical for immediate adoption.
1. Detail-oriented Teeth Aligner
This module extracts high-fidelity features using a dual-branch approach to handle the complex geometry of the mouth:
- DRM-Conv (Dense Remap Convolution): Uses a space-to-depth transformation to preserve fine-grained details and edges that standard convolutions might blur.
- PKConv (Polymorphic Kernel Convolution): An adaptive sampling mechanism that adjusts to the complex, varying shapes of teeth and the way they are occluded by lips during speech.
2. Memory-guided Teeth Stabilizer
To prevent flickering, the model uses a dual-stream memory system inspired by post-orthodontic retainers:
- Long-term Memory (LTM): A 30-frame buffer that stores high-quality "exemplars" of the subject's teeth structure, ensuring the global shape stays consistent throughout the video.
- Short-term Memory (STM): A 4-frame sliding window using ConvLSTM to capture local temporal dynamics, ensuring smooth transitions between adjacent frames.
- Cross-Attention Fusion: This mechanism dynamically balances the global stability of the LTM with the local motion requirements of the STM based on the current frame's quality.
Datasets and Training
The researchers utilized two primary sources for training and evaluation:
- HDTF (High-Definition Talking Face): A standard benchmark containing 15.8 hours of video across 362 identities, chosen for its clear teeth exposure.
- Self-built News Anchor Dataset: A new high-resolution (720P/1080P) dataset featuring 24.7 hours of news anchors. This was specifically curated because anchors have professional speaking speeds and high "teeth visibility," providing the model with better ground-truth data for rigid structures.
Key Insights and Results
The paper reframes teeth generation as a dynamic inpainting task under constant occlusion. Because teeth are rigid, the model shouldn't have to "re-imagine" them every frame. By caching high-quality features in the LTM, the model can "borrow" details from previous clear frames to fill in occluded ones.
Testing OrthoNet against five SOTA baselines showed significant improvements:
- Teeth Stability Metric (TSM): Improved by 9.8% to 15.7% across different models, indicating much less flickering and "morphing."
- Edge Clarity Index (ECI): Increased by 0.2–0.3 points, reflecting sharper teeth edges and more realistic textures.
- Qualitative Improvement: Visual comparisons show that OrthoNet successfully eliminates "hallucinated" lower teeth and restores the blurred gaps between teeth that plague other models.
Limitations and Future Work
While OrthoNet significantly improves the mouth region, its impact on global video metrics (like FVD) is more modest, as it doesn't address other facial artifacts like hair or background jitter. Additionally, the paper does not deeply explore performance under extreme profile views where perspective distortion of rigid structures is most challenging. Future work may involve distilling these features into a lighter-weight architecture to reduce the inference overhead introduced by the dual-memory buffers.
Summary updated