TL;DR
TaoMate-FVSR is a novel audio-guided facial video restoration framework that fixes "mushy" mouth details and poor lip-syncing in low-quality or AI-generated videos. By combining a Video Diffusion Transformer with speech-driven guidance, it reconstructs plausible teeth and lip movements while maintaining the subject's identity and temporal consistency.
What Problem Does This Paper Address?
Restoring faces in video is notoriously difficult, particularly the mouth region. Traditional super-resolution methods often struggle when the source material is heavily compressed or AI-generated (AIGC). In these cases, the "fine details" aren't just missing—they might be fundamentally wrong, such as "hallucinated" teeth that look like a single white block or lip movements that don't match the spoken audio.
If a model tries too hard to preserve the source, it preserves these structural errors. If it regenerates the mouth from scratch, it risks "identity drift," where the person starts looking like someone else. TaoMate-FVSR tackles this by using audio as a reliable "ground truth" for what the mouth should be doing, even when the visual evidence is unreliable.
Methods
The researchers built TaoMate-FVSR on a Video Diffusion Transformer (DiT) operating in the latent space of a causal video autoencoder. The framework introduces several key innovations:
- Source–Noise Reconstruction: Instead of starting from pure noise (standard diffusion) or strictly following the input (standard super-resolution), the model mixes the input video latents with Gaussian noise. This "noise-mixing" gives the model the freedom to ignore corrupted source details and regenerate them, while still being anchored by the original head pose and motion.
- Structured Conditioning: To keep the person looking like themselves over long videos, the model uses a "reference canvas." This canvas has dedicated slots for sparse appearance references (high-quality images of the person) and continuous history (previously generated frames).
- Two-Stage Training:
- Visual Restoration: The model first learns to recover details from degraded images and videos.
- Audio-Guided Lip Correction: The model is then trained on "dubbed" videos where the mouth movement doesn't match the audio. This forces the model to prioritize the audio cues (processed via a frozen HuBERT encoder) over the incorrect visual mouth movements.
Datasets
The authors evaluated their model using a mix of established talking-head benchmarks and modern AI-generated content:
- Standard Benchmarks: 150 videos from MEAD, HDTF, and TalkVid.
- AIGC Footage: Native clips from the MiniMax-H3 text-to-video model were used to test real-world performance on generated content.
- Internal Data: Used for training the lip-correction stage with dubbed sources.
Key Insights and Ideas
- Audio as a Structural Prior: When pixels are blurry, audio provides the most reliable information about the shape of the mouth and the visibility of teeth.
- The "Freedom" to Fix: By initializing the diffusion process with a mix of source latents and noise, the model can "overwrite" bad data. This is a significant departure from models that treat restoration as a purely additive process.
- Identity Anchoring: Using a fixed appearance reference (like a high-quality first frame) in every inference window prevents the "flickering" or changing of facial features that often plagues long-form video generation.
Key Results
TaoMate-FVSR outperformed several state-of-the-art baselines (including SparkVSR and FlashVSR) across multiple metrics:
- Perceptual Quality: It achieved the lowest LPIPS (0.0793) and DISTS (0.0858) scores, indicating the results look most similar to high-quality ground truth.
- Synchronization: It reached the best lip-sync scores (Sync-C/D), proving that the audio guidance effectively corrected mouth movements.
- Qualitative Improvements: On AI-generated videos, the model successfully resolved distinct tooth boundaries and sharp lip contours where other models produced blurry or fused results.
Limitations and Future Work
The authors note that the reconstructed details are "generative estimates." While they look plausible and match the audio, they are not a literal reconstruction of the original person's specific dental anatomy.
Furthermore, the paper highlights the ethical implications of this technology. Because the model can perform high-quality visual dubbing (changing what a person's mouth says to match new audio), the authors emphasize the need for appropriate permissions and the disclosure of edited content to mitigate the risk of impersonation or "deepfakes." Future work may involve further refining the efficiency of the diffusion process for real-time applications.
Summary updated