Abstract
Dubbing is a technique for translating video content from one language to another. However, state-of-the-art visual dubbing techniques directly copy facial expressions from source to target actors without considering identity-specific idiosyncrasies such as a unique type of smile. We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages. At the heart of our approach is the concept of motion style, in particular for facial expressions, i.e., the person-specific expression change that is yet another essential factor beyond visual accuracy in face editing applications. Our method is based on a recurrent generative adversarial network that captures the spatiotemporal co-activation of facial expressions, and enables generating and modifying the facial expressions of the target actor while preserving their style. We train our model with unsynchronized source and target videos in an unsupervised manner using cycle-consistency and mouth expression losses, and synthesize photorealistic video frames using a layered neural face renderer. Our approach generates temporally coherent results, and handles dynamic backgrounds. Our results show that our dubbing approach maintains the idiosyncratic style of the target actor better than previous approaches, even for widely differing source and target actors.
Introduction
Visual dubbing—altering a video so an actor's mouth movements match a different language—is notoriously difficult. Traditional methods often suffer from the "uncanny valley" effect because they typically perform a direct expression transfer: they copy the source actor's (the dubber's) facial movements onto the target actor.
However, every person has a "signature style"—the specific way they smile, squint, or move their lips. If you map one actor's expressions directly onto another, the result often looks like the target is "wearing" the source's face, breaking the actor's brand and identity. This research introduces a style-preserving dubbing pipeline that translates the content of speech while maintaining the unique idiosyncrasies of the target performer.
Methodology
The proposed approach utilizes a three-stage pipeline to achieve photorealistic, style-preserving results.
1. 3D Face Modeling
The process begins by reconstructing the 3D geometry, pose, and expression parameters of both the source and target actors from monocular (single-camera) video using a multilinear face model. This allows the system to work in a low-dimensional parameter space rather than directly on pixels, making the model more robust and data-efficient.
2. Style Translation Network
Instead of copying parameters directly, the system employs a Recurrent Generative Adversarial Network (GAN). This network is designed to capture the spatiotemporal co-activation of facial expressions.
- Temporal Dynamics: The architecture incorporates Long Short-Term Memory (LSTM) units to ensure that generated expressions are smooth and temporally coherent.
- Unsupervised Learning: The network is trained using cycle-consistency loss, ensuring a mapping from Source → Target → Source returns the original expression.
- Cosine Mouth Expression Loss: This novel loss function encourages the mouth to open and close at the right times to match the dubber without forcing the absolute magnitude of the opening to match, allowing the target actor to maintain their own "mouth-opening style."
3. Layered Neural Face Renderer
Finally, a neural network converts the translated 3D parameters back into photorealistic pixels. Unlike previous work that assumed static backgrounds, this "layered" approach uses a soft face mask to composite the new face onto the original video, handling dynamic backgrounds seamlessly.
Key Results and Insights
The researchers evaluated their method on a diverse dataset of 11 source and 12 target sequences, totaling over 50 minutes of video across multiple languages (English, German, Indonesian, and Kannada).
- Motion Style as Identity: The study demonstrates that "motion style" is as essential to identity as facial geometry. A dub is only successful if the target actor still "feels" like themselves.
- User Preference: In a web-based user study (n=50), the style-preserving approach received a "natural" rating of 60%, compared to just 8% for naive direct-transfer dubbing.
- Comparison to State-of-the-Art: The method outperformed previous systems like VDub and Deep Video Portraits. In pairwise comparisons, users preferred the style-preserving results 73% of the time over the VDub approach.
- Qualitative Success: The system successfully maintained idiosyncratic behaviors, such as "near-closed eyes" for actors who naturally squint while speaking, even when the dubber had wide-open eyes.
Conclusion and Future Work
While the approach generates temporally coherent results and handles dynamic backgrounds, it is sensitive to the quality of the initial 3D reconstruction. Extreme lighting or sharp head turns can cause tracking failures. Furthermore, while efficient (requiring only ~2 minutes of target video), the model must still be trained for each specific target actor. Future research may explore "zero-shot" style preservation that generalizes across different individuals without actor-specific training.
Summary updated