Neural style-preserving visual dubbing

Hyeongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo Beeler, Christian Richardt, Christian Theobalt

TL;DR

This paper introduces a novel visual dubbing pipeline that modifies a target actor's lip movements to match a new audio track while preserving their unique facial "style." By using a recurrent generative adversarial network (GAN) trained in an unsupervised manner, the system ensures that a dubbed performance still looks like the original actor’s idiosyncratic behavior rather than a robotic copy of the dubber.

What Problem Does This Paper Address?

Visual dubbing—altering a video so an actor's mouth movements match a different language—is notoriously difficult. Traditional methods often suffer from the "uncanny valley" effect. The core issue, according to the authors, is that state-of-the-art techniques typically perform a direct expression transfer: they copy the source actor's (the dubber's) facial movements onto the target actor.

However, every person has a "signature style"—the specific way they smile, squint, or move their lips. If you map Robert De Niro’s expressions directly onto Sylvester Stallone, the result looks like Stallone "wearing" De Niro’s face, which breaks the actor's brand and identity. This paper tackles the challenge of style-preserving dubbing: translating the content of the speech while maintaining the idiosyncrasies of the target performer.

Methods

The authors propose a three-stage pipeline:

  1. 3D Face Modeling: They first reconstruct the 3D geometry, pose, and expression parameters of both the source and target actors from monocular (single-camera) video using a multilinear face model.
  2. Style Translation Network: This is the heart of the paper. Instead of copying parameters directly, they use a Recurrent Generative Adversarial Network (GAN).
  3. Temporal Dynamics: They incorporate Long Short-Term Memory (LSTM) units to ensure the expressions are smooth and temporally coherent over time.
  4. Unsupervised Learning: The network is trained using cycle-consistency loss (ensuring a mapping from Source $\to$ Target $\to$ Source returns the original expression).
  5. Cosine Mouth Expression Loss: A novel loss function that encourages the mouth to open and close at the right times (matching the dubber) without forcing the magnitude of the opening to match, allowing the target actor to maintain their own "mouth-opening style."
  6. Layered Neural Face Renderer: Finally, they use a neural network to convert the translated 3D parameters back into photorealistic pixels. Unlike previous work that assumed static backgrounds, this "layered" approach uses a soft face mask to composite the new face onto the original video, handling dynamic backgrounds seamlessly.

Datasets

The researchers evaluated their method on a diverse dataset of 11 source and 12 target sequences, totaling over 50 minutes of video.

  • Languages: Included English, German, Indonesian, and Kannada.
  • Styles: The sequences featured various expressions, including "neutral + smile," "angry," "sarcastic," and "squint."
  • Source Material: Included high-quality captures and public domain footage (e.g., President Obama).

Key Insights and Ideas

  • Motion Style as a Feature: The authors argue that "motion style" is as essential to identity as facial geometry. A dub is only successful if the target actor still "feels" like themselves.
  • Parameter Space vs. Image Space: By performing the style translation in a low-dimensional 3D parameter space (rather than directly on pixels), the model is much more robust and requires significantly less training data (only ~2 minutes of video) compared to pure image-to-image translation methods like CycleGAN.
  • The Power of Cosine Loss: Using cosine similarity for mouth expressions allows the model to capture the direction of movement (opening vs. closing) while ignoring the scale, which is exactly what is needed to preserve individual style (e.g., one person might have a much wider "natural" smile than another).

Key Results

  • User Preference: In a web-based user study (n=50), the style-preserving approach received a "natural" rating of 60%, compared to just 8% for naïve dubbing (direct transfer).
  • Comparison to SOTA: The method outperformed previous state-of-the-art systems like VDub and Deep Video Portraits. In pairwise comparisons, users preferred the style-preserving results 73% of the time over the VDub approach.
  • Qualitative Success: The system successfully maintained "near-closed eyes" for actors who naturally squint while speaking, even when the dubber had wide-open eyes. It also handled complex scenarios like German-to-English dubbing with dynamic, moving backgrounds.

Limitations and Future Work

  • Extreme Expressions: The model can struggle if the source and target distributions are too different. For example, an extreme "one-eyed twitch" from a dubber might produce implausible artifacts on the target actor.
  • Reconstruction Failures: Like all model-based approaches, it is sensitive to the quality of the initial 3D reconstruction. Extreme lighting or very sharp head turns can cause the tracking to fail.
  • Data Requirements: While efficient, the model still requires a few minutes of video for each specific target actor to learn their style. Future work could explore "zero-shot" style preservation that generalizes across different people.

Summary updated

Abstract

Dubbing is a technique for translating video content from one language to another. However, state-of-the-art visual dubbing techniques directly copy facial expressions from source to target actors without considering identity-specific idiosyncrasies such as a unique type of smile. We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages. At the heart of our approach is the concept of motion style, in particular for facial expressions, i.e., the person-specific expression change that is yet another essential factor beyond visual accuracy in face editing applications. Our method is based on a recurrent generative adversarial network that captures the spatiotemporal co-activation of facial expressions, and enables generating and modifying the facial expressions of the target actor while preserving their style. We train our model with unsynchronized source and target videos in an unsupervised manner using cycle-consistency and mouth expression losses, and synthesize photorealistic video frames using a layered neural face renderer. Our approach generates temporally coherent results, and handles dynamic backgrounds. Our results show that our dubbing approach maintains the idiosyncratic style of the target actor better than previous approaches, even for widely differing source and target actors.
BibTeX
@article{kim2019neural,
  author = {Hyeongwoo Kim and Mohamed Elgharib and Michael Zollhöfer and Hans-Peter Seidel and Thabo Beeler and Christian Richardt and Christian Theobalt},
  title = {{Neural style-preserving visual dubbing}},
  journal = {ACM Trans. Graphics},
  year = {2019},
  url = {https://openreview.net/forum?id=TPPsRAH44FO},
}

Loading related content...

Figure