Outstanding Orthodontist: No More Artifactual Teeth in Talking Face

Zibo Su, Ziqi Zhang, Kun Wei, Xu Yang, Cheng DengIJCAI 2025 · DOI · OpenReview

TL;DR

OrthoNet is a novel "plug-and-play" framework designed to fix one of the most persistent artifacts in talking face synthesis: distorted, flickering, and unrealistic teeth. By combining a detail-preserving feature extractor with a dual-memory stabilization system, it ensures that teeth remain rigid, clear, and temporally consistent, even during complex lip movements and occlusions.

The Challenge: Crossing the Dental Uncanny Valley

While modern audio-driven Talking Face Synthesis (TFS) has made massive strides in lip-syncing and emotional expression, the "teeth region" remains a major hurdle. Unlike skin or lips, teeth are rigid structures that should not deform. However, current models often treat the mouth as a generic deformable area, leading to two primary issues:

  1. Temporal Inconsistency: Teeth appear to "grow," "shrink," or flicker between frames because models generate each frame independently or with limited context.
  2. Generation Hallucinations: Due to frequent occlusion by the lips, models effectively have to "inpaint" the teeth. Without strong priors, this results in blurred textures, missing gaps, or "hallucinated" extra teeth.
Comparison between real video and state-of-the-art Audio2Video generation (Hallo2), highlighting the distorted teeth artifacts.

Methods: The OrthoNet Architecture

The authors propose OrthoNet, which acts as a virtual "orthodontist" for existing Audio2Video models. It is designed to be added to the end of existing state-of-the-art (SOTA) pipelines (like AniPortrait or EchoMimic), making it highly practical for immediate adoption.

The OrthoNet architecture, featuring the Detail-oriented Teeth Aligner and the Memory-guided Teeth Stabilizer.

1. Detail-oriented Teeth Aligner

This module extracts high-fidelity features using a dual-branch approach to handle the complex geometry of the mouth:

  • DRM-Conv (Dense Remap Convolution): Uses a space-to-depth transformation to preserve fine-grained details and edges that standard convolutions might blur.
  • PKConv (Polymorphic Kernel Convolution): An adaptive sampling mechanism that adjusts to the complex, varying shapes of teeth and the way they are occluded by lips during speech.

2. Memory-guided Teeth Stabilizer

To prevent flickering, the model uses a dual-stream memory system inspired by post-orthodontic retainers:

  • Long-term Memory (LTM): A 30-frame buffer that stores high-quality "exemplars" of the subject's teeth structure, ensuring the global shape stays consistent throughout the video.
  • Short-term Memory (STM): A 4-frame sliding window using ConvLSTM to capture local temporal dynamics, ensuring smooth transitions between adjacent frames.
  • Cross-Attention Fusion: This mechanism dynamically balances the global stability of the LTM with the local motion requirements of the STM based on the current frame's quality.

Datasets and Training

The researchers utilized two primary sources for training and evaluation:

  • HDTF (High-Definition Talking Face): A standard benchmark containing 15.8 hours of video across 362 identities, chosen for its clear teeth exposure.
  • Self-built News Anchor Dataset: A new high-resolution (720P/1080P) dataset featuring 24.7 hours of news anchors. This was specifically curated because anchors have professional speaking speeds and high "teeth visibility," providing the model with better ground-truth data for rigid structures.

Key Insights and Results

The paper reframes teeth generation as a dynamic inpainting task under constant occlusion. Because teeth are rigid, the model shouldn't have to "re-imagine" them every frame. By caching high-quality features in the LTM, the model can "borrow" details from previous clear frames to fill in occluded ones.

Testing OrthoNet against five SOTA baselines showed significant improvements:

  • Teeth Stability Metric (TSM): Improved by 9.8% to 15.7% across different models, indicating much less flickering and "morphing."
  • Edge Clarity Index (ECI): Increased by 0.2–0.3 points, reflecting sharper teeth edges and more realistic textures.
  • Qualitative Improvement: Visual comparisons show that OrthoNet successfully eliminates "hallucinated" lower teeth and restores the blurred gaps between teeth that plague other models.

Limitations and Future Work

While OrthoNet significantly improves the mouth region, its impact on global video metrics (like FVD) is more modest, as it doesn't address other facial artifacts like hair or background jitter. Additionally, the paper does not deeply explore performance under extreme profile views where perspective distortion of rigid structures is most challenging. Future work may involve distilling these features into a lighter-weight architecture to reduce the inference overhead introduced by the dual-memory buffers.

Summary updated

Abstract

Audio-driven talking face synthesis (TFS) enables the creation of realistic speaking videos by combining a single facial image with a speech audio clip. Unlike other facial features that naturally deform during speech, teeth represent unique rigid structures whose shape and size should remain constant throughout the video sequence. However, current methods often produce temporal inconsistencies and artifacts in the teeth region, resulting in a less realistic appearance of the generated videos. To address this, we propose OrthoNet, a plug-and-play framework designed to eliminate unrealistic teeth effects in audio-driven TFS. Our method introduces a Detail-oriented Teeth Aligner module, designed to preserve teeth details and adapt to their shape. It works with a Memory-guided Teeth Stabilizer that integrates a long-term memory bank for global teeth structure and a short-term memory module for local temporal dynamics. Through this framework, OrthoNet acts like an orthodontist for existing Audio2Video methods, ensuring that teeth maintain natural rigidity and temporal consistency even under varying degrees of teeth occlusion. Extensive experiments demonstrate that our method makes the teeth in generated videos appear more natural during speech, significantly enhancing the temporal consistency and structural stability of audio-driven video generation.
BibTeX
@inproceedings{su2025outstanding,
  author = {Zibo Su and Ziqi Zhang and Kun Wei and Xu Yang and Cheng Deng},
  title = {{Outstanding Orthodontist: No More Artifactual Teeth in Talking Face}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/211},
}
Figure