TaoMate-FVSR: Audio-Guided Facial Video Restoration

Xinyang Li, Zeran Xu, Jingna Sun, Qijun Gan, Chenwei Zhang, Meiguang Jin, Junfeng MaarXiv 2026 · arXiv

TL;DR

TaoMate-FVSR is a novel audio-guided facial video restoration framework that fixes "mushy" mouth details and poor lip-syncing in low-quality or AI-generated videos. By combining a Video Diffusion Transformer with speech-driven guidance, it reconstructs plausible teeth and lip movements while maintaining the subject's identity and temporal consistency.

What Problem Does This Paper Address?

Restoring faces in video is notoriously difficult, particularly the mouth region. Traditional super-resolution methods often struggle when the source material is heavily compressed or AI-generated (AIGC). In these cases, the "fine details" aren't just missing—they might be fundamentally wrong, such as "hallucinated" teeth that look like a single white block or lip movements that don't match the spoken audio.

If a model tries too hard to preserve the source, it preserves these structural errors. If it regenerates the mouth from scratch, it risks "identity drift," where the person starts looking like someone else. TaoMate-FVSR tackles this by using audio as a reliable "ground truth" for what the mouth should be doing, even when the visual evidence is unreliable.

Methods

The researchers built TaoMate-FVSR on a Video Diffusion Transformer (DiT) operating in the latent space of a causal video autoencoder. The framework introduces several key innovations:

  • Source–Noise Reconstruction: Instead of starting from pure noise (standard diffusion) or strictly following the input (standard super-resolution), the model mixes the input video latents with Gaussian noise. This "noise-mixing" gives the model the freedom to ignore corrupted source details and regenerate them, while still being anchored by the original head pose and motion.
  • Structured Conditioning: To keep the person looking like themselves over long videos, the model uses a "reference canvas." This canvas has dedicated slots for sparse appearance references (high-quality images of the person) and continuous history (previously generated frames).
  • Two-Stage Training:
    1. Visual Restoration: The model first learns to recover details from degraded images and videos.
  • Audio-Guided Lip Correction: The model is then trained on "dubbed" videos where the mouth movement doesn't match the audio. This forces the model to prioritize the audio cues (processed via a frozen HuBERT encoder) over the incorrect visual mouth movements.
The TaoMate-FVSR pipeline showing the two-stage training process and the integration of audio and reference conditioning.

Datasets

The authors evaluated their model using a mix of established talking-head benchmarks and modern AI-generated content:

  • Standard Benchmarks: 150 videos from MEAD, HDTF, and TalkVid.
  • AIGC Footage: Native clips from the MiniMax-H3 text-to-video model were used to test real-world performance on generated content.
  • Internal Data: Used for training the lip-correction stage with dubbed sources.

Key Insights and Ideas

  • Audio as a Structural Prior: When pixels are blurry, audio provides the most reliable information about the shape of the mouth and the visibility of teeth.
  • The "Freedom" to Fix: By initializing the diffusion process with a mix of source latents and noise, the model can "overwrite" bad data. This is a significant departure from models that treat restoration as a purely additive process.
  • Identity Anchoring: Using a fixed appearance reference (like a high-quality first frame) in every inference window prevents the "flickering" or changing of facial features that often plagues long-form video generation.

Key Results

TaoMate-FVSR outperformed several state-of-the-art baselines (including SparkVSR and FlashVSR) across multiple metrics:

  • Perceptual Quality: It achieved the lowest LPIPS (0.0793) and DISTS (0.0858) scores, indicating the results look most similar to high-quality ground truth.
  • Synchronization: It reached the best lip-sync scores (Sync-C/D), proving that the audio guidance effectively corrected mouth movements.
  • Qualitative Improvements: On AI-generated videos, the model successfully resolved distinct tooth boundaries and sharp lip contours where other models produced blurry or fused results.
Figure 3. Facial detail and articulation in AIGC scenes. Four MiniMax-H3 [41] clips are shown with matched-time head crops located by orange boxes. Left: restoration under original audio. Right: FlashVSR Full (top) or SparkVSR (bottom) is followed by the arrow-marked dubbing methods; Input and SR-only outputs provide visual controls. Ours+

Limitations and Future Work

The authors note that the reconstructed details are "generative estimates." While they look plausible and match the audio, they are not a literal reconstruction of the original person's specific dental anatomy.

Furthermore, the paper highlights the ethical implications of this technology. Because the model can perform high-quality visual dubbing (changing what a person's mouth says to match new audio), the authors emphasize the need for appropriate permissions and the disclosure of edited content to mitigate the risk of impersonation or "deepfakes." Future work may involve further refining the efficiency of the diffusion process for real-time applications.

Summary updated

Abstract

Low-quality facial videos suffer from missing fine details, while AI-generated videos may additionally contain implausible teeth and lip movements that do not match the speech. Restoring the mouth region is particularly challenging: enhancing existing details can preserve structural errors, whereas regenerating them risks changing facial identity and articulation. We propose an audio-guided facial video restoration framework that reconstructs unreliable mouth details and aligns lip movements with the speech while preserving facial identity and temporal consistency. Built on a video diffusion transformer, our framework mixes input video latents with noise to retain useful source information while allowing corrupted details to be regenerated. Audio guides lip movements, reference images constrain facial appearance, and previously generated frames promote temporal consistency. We train the model in two stages, first learning visual restoration from degraded images and videos, then learning audio-guided lip correction from degraded dubbed videos. The resulting model supports both restoration with the original audio and visual dubbing with replacement audio. On talking-head restoration benchmarks, our method achieves the lowest LPIPS and DISTS and the best lip synchronization scores among the evaluated restoration baselines. Qualitative results on AI-generated videos further show improvements in facial detail and mouth-region appearance, including clearer tooth boundaries and lip contours.
BibTeX
@misc{li2026taomate,
  author = {Xinyang Li and Zeran Xu and Jingna Sun and Qijun Gan and Chenwei Zhang and Meiguang Jin and Junfeng Ma},
  title = {{TaoMate-FVSR: Audio-Guided Facial Video Restoration}},
  year = {2026},
  eprint = {2611.03393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03393},
}
Figure