A mouth full of words: Visually consistent acoustic redubbing

Sarah L. Taylor, Barry-John Theobald, Iain A. MatthewsICASSP 2015 · OpenReview

Abstract

This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes (Taylor et al., 2012). For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, many-to-one, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.

Introduction

Acoustic redubbing—the process of replacing a video's audio track—is a critical task for film translation and content editing. Traditionally, this is a labor-intensive manual process requiring scriptwriters to find words that visually match the speaker's lip movements and actors to time their delivery perfectly. The core technical challenge is the "many-to-many" mapping between phonemes (sounds) and visemes (visual shapes).

Conventional models rely on "static visemes," which group multiple phonemes into fixed mouth shapes. However, these models often fail to account for coarticulation—the influence of preceding and succeeding sounds on current mouth position. This rigidity limits the linguistic variety available for redubbing. This research proposes a shift toward "dynamic visemes" to capture the fluid nature of speech gestures.

Methodology

The proposed system models speech as a sequence of movements rather than static poses. The pipeline begins with feature extraction using Active Appearance Models (AAMs) to track facial motion in a 20D vector space. The video is then segmented into non-overlapping gestures based on salient points, such as the peak of a lip opening.

Mapping phoneme sequences to dynamic visemes using AAM parameters

These gestures are clustered into 150 dynamic viseme (DV) classes. Each class represents a probability distribution of possible phoneme strings. A directed acyclic graph is constructed where nodes represent unique phoneme sequences for each gesture, using boundary context labels to ensure linguistic validity during transitions. The resulting phoneme graph is searched against the CMU Pronouncing Dictionary and ranked using a bigram language model trained on the Open American National Corpus. Finally, the selected text is synthesized and retimed to match the original gesture durations. The system was trained and evaluated using the KB-2K dataset, featuring 8 hours of high-definition video of a single actor.

Results and Discussion

The dynamic viseme approach was compared against two static viseme mappings: Jeffers & Barley (JB) and Parke & Waters (PW). The results highlight a significant increase in flexibility. As shown in the figure below, the DV approach generates a vastly larger number of word sequences for a given phoneme length compared to static models.

Comparison of word sequence generation flexibility between dynamic visemes (blue) and static models (green, red)

While static models frequently failed to find any valid word sequences (58% failure for PW and 17% for JB), the DV model consistently provided options. This demonstrates that the inherent ambiguity of visual speech, often a hurdle for speech recognition, can be leveraged as a feature for redubbing. Qualitatively, the system successfully redubbed phrases like "clean swatches" into visually synchronized alternatives like "likes swats" or "need no pots."

Conclusion

By modeling speech as dynamic gestures, this method provides a robust framework for automatic redubbing. Future work will focus on improving the naturalness of generated sentences through more sophisticated language models and extending the system to handle diverse speakers, head poses, and speaking styles. This research underscores the importance of modeling the temporal dynamics of visual speech for both synthesis and recognition tasks.

Summary updated

Abstract

This paper introduces a method for automatic redubbing of video that exploits the many-to-many mapping of phoneme sequences to lip movements modelled as dynamic visemes (Taylor et al., 2012). For a given utterance, the corresponding dynamic viseme sequence is sampled to construct a graph of possible phoneme sequences that synchronize with the video. When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker. We demonstrate that traditional, many-to-one, static visemes lack flexibility for this application as they produce significantly fewer word sequences. This work explores the natural ambiguity in visual speech and offers insight for automatic speech recognition and the importance of language modeling.
BibTeX
@inproceedings{taylor2015mouth,
  author = {Sarah L. Taylor and Barry-John Theobald and Iain A. Matthews},
  title = {{A mouth full of words: Visually consistent acoustic redubbing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=SSePJd2Xz79},
}
Figure