Speaker-adaptive visual speech synthesis in the HMM-framework

Dietmar Schabus, Michael Pucher, Gregor HoferInterspeech 2012

TL;DR

This paper tackles the challenge of creating realistic talking face animations for new speakers with limited data. They adapt a successful acoustic speech synthesis technique called speaker-adaptive training (SAT) to the visual domain, showing it outperforms traditional speaker-dependent methods when training data is scarce. This opens doors for more personalized and efficient visual speech synthesis.

What Problem Does This Paper Address?

Creating realistic audio-visual speech synthesis, where a synthesized voice is paired with a matching animated talking face, is a complex task. While acoustic speech synthesis has seen significant advancements, generating convincing visual speech animations, especially for new speakers with little available data, remains a hurdle. Traditional methods often require large amounts of data per speaker, making it impractical for many applications. This paper aims to bridge this gap by adapting a proven technique from acoustic speech synthesis – speaker-adaptive training (SAT) – to the visual domain, enabling the creation of personalized facial animations with significantly less data.

Methods

The core of this paper lies in adapting the speaker-adaptive training (SAT) framework, previously successful in acoustic speech synthesis, to visual speech. Here's a breakdown of their approach:

  • 3D Generative Model: They utilize a 3D generative model for facial animation, which offers more flexibility than image-based methods.
  • Hidden Markov Models (HMMs): HMMs are employed to model the visual speech features. Specifically, they use context-dependent, left-to-right, hidden semi-Markov models (HSMMs) to capture both visual features and their durations.
  • Principal Component Analysis (PCA): To reduce the dimensionality of the 3D facial marker data (originally 99 dimensions), they apply PCA. This helps in creating a more manageable feature space (reduced to 30 dimensions) for HMM training.
  • Speaker-Adaptive Training (SAT):
    • Average Voice Training: A "background" or "average" voice model is trained using data from multiple speakers. This model captures general visual speech patterns.
  • Speaker Adaptation: For a new target speaker, this average model is then adapted using a small amount of data from that specific speaker. This adaptation process fine-tunes the general model to the individual's unique facial movements.
  • Speaker-Dependent Training: As a baseline for comparison, they also train traditional speaker-dependent models, where an HMM is trained solely on data from the target speaker.

Datasets

  • Synchronous Acoustic and 3D Facial Marker Data: The authors introduce a newly recorded corpus consisting of synchronous acoustic and 3D facial marker data. This corpus includes data from three speakers of Austrian German, each reading 223 phonetically balanced utterances. The 3D facial data captures the positions of 41 reflective markers on the speakers' faces at 100Hz.
  • HTK Quin-phone Full-Context Label Files: This data provides precise temporal phone borders, crucial for aligning acoustic and visual information.

Key Insights and Ideas

  • Visual SAT is Viable: The paper successfully demonstrates that the principles of speaker-adaptive training, proven effective in acoustic speech synthesis, can be effectively transferred to the visual domain for facial animation.
  • Data Efficiency is Key: The primary motivation and success of SAT in this context is its ability to produce good results with significantly less training data for a new speaker compared to speaker-dependent methods. This is crucial for practical applications where acquiring large datasets per speaker is often infeasible.
  • Perceptual Preference: Subjective evaluations reveal that speaker-adaptive models, especially when using a small amount of adaptation data, are often preferred over speaker-dependent models trained on the same limited data. This suggests that the "average voice" provides a better starting point, leading to more natural-sounding animations.
  • Objective Metrics Can Be Misleading: The paper highlights the challenges of relying solely on objective metrics (like RMSE) for evaluating visual speech synthesis. Temporal misalignments, even subtle ones, can lead to high errors despite the overall movement being "correct," underscoring the importance of perceptual studies.

Key Results

  • Perceptual Preference: In a perceptual experiment with 28 subjects, the speaker-adaptive model with a small amount of adaptation data ("adapt small") was statistically significantly preferred over the speaker-dependent model trained on the same small data ("sd small").
  • Trend Towards SAT: While the direct comparison between speaker-dependent ("sd") and speaker-adaptive ("adapt") models (using a larger adaptation set) was not statistically significant, there was a trend favoring the speaker-dependent model. However, when data was limited, the adaptive approach showed its strength.
  • Objective Results: Objective evaluation using RMSE showed that "sd small" was slightly worse than other methods, likely due to temporal misalignments. However, the paper emphasizes that these objective results were not very informative for comparing the methods.

Limitations and Future Work

  • Limited Speaker Data: The current study used a relatively small number of speakers (three) for the initial corpus.
  • Speaker Similarity: The paper acknowledges that speaker similarity, or how closely synthesized data mimics specific speaker characteristics, will be investigated further.
  • New Multi-Speaker Database: The authors are currently recording a larger multi-speaker audio-visual database of different dialects of Austrian German, which will allow for more extensive testing and generalization.
  • More Speakers and Data: Future work will focus on applying the developed methods to more speakers and larger datasets.

Summary updated

BibTeX
@inproceedings{schabus2012speaker,
  author = {Dietmar Schabus and Michael Pucher and Gregor Hofer},
  title = {{Speaker-adaptive visual speech synthesis in the HMM-framework}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure