Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Salil Deena, Shaobo Hou, Aphrodite GalataICMI-MLMI 2010 · DOI

TL;DR

This paper introduces a novel approach to creating realistic talking-head animations driven by speech. They achieve this by using a sophisticated statistical model that captures the complex interplay between speech sounds and facial movements, specifically focusing on how sounds influence each other (coarticulation). This leads to more natural and dynamic facial animations compared to previous methods.

What Problem Does This Paper Address?

Creating realistic facial animations that sync with speech is a challenging task. The core problem is accurately modeling the relationship between audio (speech sounds) and visual (facial movements). This relationship isn't one-to-one; a single speech sound (phoneme) can correspond to multiple visual expressions (visemes), and importantly, the visual appearance of a sound is heavily influenced by the sounds that come before and after it. This phenomenon, known as coarticulation, is crucial for natural speech but difficult to model. Previous methods often struggled to capture these dynamics, leading to animations that felt stiff or unnatural.

Methods

The authors propose a Switching Shared Gaussian Process Dynamical Model (SSGPDM). Here's a breakdown of their approach:

  • Gaussian Processes (GPs): GPs are used as a flexible, non-parametric way to model complex, non-linear relationships. They allow the model to learn from data without making strong assumptions about the underlying functional forms.
  • Shared Latent Space: Both audio and visual data are mapped into a common "latent space." This shared space helps to find correlations and dependencies between the two modalities.
  • Switching States: To capture the varying dynamics of speech (different sounds and their influences), the model incorporates "switching states." These states are analogous to different "modes" of facial articulation.
  • Variable Length Markov Models (VLMMs): To automatically discover these switching states, they train VLMMs on phonetic labels from the speech data. VLMMs are adept at learning patterns in sequential data where the relevant context length can vary.
  • Coarticulation Modeling: The SSGPDM explicitly models both "carryover coarticulation" (how a sound influences subsequent sounds) and "anticipatory coarticulation" (how upcoming sounds influence the current one) by leveraging the VLMM states and a smoothness constraint during synthesis.

Datasets

  • LIPS corpus: This is the primary dataset used for training and evaluation. It consists of high-quality video sequences of a female subject speaking sentences, along with corresponding audio and phonetic annotations.

Key Insights and Ideas

  • Coarticulation is Key: The paper strongly emphasizes that explicitly modeling coarticulation is essential for natural visual speech synthesis. Ignoring it leads to less realistic animations.
  • VLMMs for State Discovery: Using VLMMs on phonetic labels is an effective way to automatically segment speech into meaningful units that capture different coarticulatory dynamics.
  • Switching Dynamics are Crucial: A single, static model is insufficient. The ability to switch between different dynamic models (one for each VLMM state) allows for more nuanced and accurate representation of speech articulation.
  • Smoothness Across States: A critical insight is the need for smooth transitions between these switching states. They introduce a smoothness constraint to prevent jarring changes in facial animation when moving from one phonetic context to another.

Key Results

  • Quantitative Improvement: The proposed SSGPDM method, particularly with sequential optimization, achieved the best quantitative results in terms of Average Mean Squared Error (AMSE) and Average Correlation Coefficient (ACC) when compared to other methods, including a simpler SSGPDM without explicit phonetic context modeling and Brand's Voice Puppetry.
  • Qualitative Evidence: Figure 4 showcases synthesized frames with corresponding phonetic labels, visually demonstrating the method's ability to generate dynamic and context-aware facial expressions.

Limitations and Future Work

  • Phonetic Labels Required: A significant limitation is the reliance on phonetic labels for both training and testing. The authors acknowledge this and suggest future work on techniques for training and synthesis using unlabelled phonetic data.
  • Human Subject Evaluation: They plan to conduct qualitative evaluations with human subjects to assess the realism and intelligibility of their animations.
  • Generalizability: While the LIPS corpus is used, further evaluation on diverse datasets and speakers would be beneficial.

Summary updated

BibTeX
@inproceedings{deena2010visual,
  author = {Salil Deena and Shaobo Hou and Aphrodite Galata},
  title = {{Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model}},
  booktitle = {ICMI-MLMI},
  year = {2010},
  doi = {10.1145/1891903.1891942},
}
Figure