VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing

Yexin Liu, Yunlong Lin, Zehan Wang, Guangxin He, Xinli Xu, Jiajun Zha, Yaofu Liu, Yueze Wang, Bowen Ping, Yuanming Li, Amie Wang, Bernard Ghanem, Bing Li, Harry YangarXiv 2026 · arXiv

TL;DR

VidDub introduces a "replacement-aware" training strategy that resolves the inherent conflict between original source mouth movements and new dubbed audio. By distilling a full-context diffusion model into just two denoising steps, it achieves state-of-the-art lip synchronization and visual quality at speeds suitable for high-throughput offline production.

What Problem Does This Paper Address?

Video dubbing—altering a speaker's lip movements to match a new audio track—is a deceptively difficult task. Most existing models are trained using self-reconstruction: they take a video and its original audio and learn to recreate the original mouth movements.

The problem arises at test time when you provide a new (replacement) audio track. The model is now caught in a tug-of-war. Its training tells it to reconstruct the mouth it sees in the source video, but the new audio demands a completely different set of articulations. This "source-mouth conflict" often leads to "mushy" lip sync or visual artifacts as the model tries to satisfy two contradictory targets.

Furthermore, many recent high-quality models rely on diffusion, which is notoriously slow due to the iterative denoising process. While "causal" models (processing frame-by-frame) are great for streaming, they often sacrifice the global temporal consistency needed for high-end offline dubbing.

Methods

The authors propose VidDub, a full-context diffusion editor that utilizes a novel three-stage training and distillation pipeline.

1. Replacement-Aware Cross-Audio Supervision

This is the core innovation. During training, VidDub uses two types of samples:

  • Paired samples: Standard original audio/video pairs where the model performs full-frame reconstruction.
  • Unpaired samples: The model is given a replacement audio track. Critically, the researchers mask out the reconstruction loss for the mouth region. Instead of trying to match the source mouth, the model is supervised solely by an audio-visual synchronization loss (Sync-C). This "decouples" the task: the source video provides the identity and pose, while the new audio exclusively drives the lips.
Figure 3: Three-stage training and distillation pipeline. Top: full-context teacher training via audio warm-up, joint editing adaptation, and replacement-aware refinement. (a) Replacement-aware training masks out source-mouth reconstruction under unpaired speech. (b) Guided trajectory regression distills the 48-step CFG-guided teacher into a two-step student. (c) Distribution matching distillation recovers visual realism in the final two-step model.

2. Full-Context Architecture

Unlike streaming-focused models, VidDub uses bidirectional self-attention across an 81-frame window. This allows every frame to "see" the entire temporal context, leading to smoother, more stable results.

Figure 2: Full-context two-step dubbing architecture. Masked source frames and a reference image preserve speaker appearance and pose, target-audio tokens drive lip articulation, and bidirectional self-attention jointly denoises the complete latent window in two steps.

3. Two-Step Distillation

To make the model fast, they distill a 48-step "teacher" model into a 2-step "student." They use Matched Trajectory Regression to teach the student to follow the teacher's denoising paths and Distribution Matching Distillation (DMD) to recover the fine visual details lost during step compression.

Datasets

The model was trained on a massive pool of over 322,000 clips from diverse sources:

  • Public Benchmarks: HDTF, VoxCeleb2, and TalkVid.
  • New Benchmark: The authors introduced VidDub-Bench, a curated set of 256 speaker-disjoint generated videos. It includes fixed same- and cross-audio pairings, structured metadata, and unified evaluation code to ensure fair comparisons across different methods.

Key Insights and Ideas

  • Masking is the "Aha!" Moment: Simply masking the mouth reconstruction loss for replacement audio yielded a massive +0.833 improvement in Sync-C (synchronization confidence). This proves that the "source-mouth conflict" was a major bottleneck in previous research.
  • Offline > Online for Quality: The paper demonstrates that for offline dubbing, bidirectional attention is superior to causal (streaming) attention. It achieves higher throughput, uses less peak memory, and results in better visual speech recognition (VSR) accuracy.
  • Distillation doesn't have to sacrifice sync: By carefully regressing trajectories from a replacement-aware teacher, the 2-step student maintains nearly identical synchronization performance to the 48-step teacher.

Key Results

VidDub sets a new bar for both speed and quality in video dubbing:

  • Inference Speed: It achieves 11.3 FPS on a single NVIDIA H20 GPU, significantly faster than competitors like LatentSync (2.2 FPS) or Lip Forcing (2.4 FPS).
  • Synchronization: On benchmarks like HDTF-30 and TalkVid-Bench, VidDub consistently achieves higher Sync-C scores than previous state-of-the-art models, even those using many more denoising steps.
  • Visual Quality: As shown in Figure 4, VidDub occupies the "top-left" frontier, combining high synchronization with low Fréchet Video Distance (FVD), indicating superior realism.
Figure 4: Cross-audio Sync-C versus FVD on four benchmarks. Dashed lines show real-video same-audio Sync-C.
Quantitative comparison showing VidDub's superior performance in both reconstruction and replacement dubbing tasks.

Limitations and Future Work

While VidDub is highly effective, the authors note a few areas for improvement:

  • Failure Modes: Like most lip-sync models, it can still suffer from occasional "teeth or tongue flicker," weak articulation in very challenging cases, or slight color drift over extremely long sequences.
  • Extreme Poses: While it handles "in-the-wild" video well, extreme head poses or rapid movements can still challenge the stability of the generated mouth region.
  • Ethical Considerations: The authors explicitly acknowledge the potential for misuse (e.g., non-consensual impersonation) and are releasing the model with a use agreement and provenance metadata to mitigate these risks.

Future work may involve further refining the "full-context" windowing for even longer videos and exploring more sophisticated ways to preserve fine-grained emotional expressions during the dubbing process.

Summary updated

Abstract

Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.
BibTeX
@misc{liu2026viddub,
  author = {Yexin Liu and Yunlong Lin and Zehan Wang and Guangxin He and Xinli Xu and Jiajun Zha and Yaofu Liu and Yueze Wang and Bowen Ping and Yuanming Li and Amie Wang and Bernard Ghanem and Bing Li and Harry Yang},
  title = {{VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing}},
  year = {2026},
  eprint = {2611.03404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03404},
}
Figure