HMM-based automatic visual speech segmentation using facial data

Utpala Musti, Asterios Toutios, Slim Ouni, Vincent Colotte, Brigitte Wrobel-Dautcourt, Marie-Odile BergerInterspeech 2010

TL;DR

This paper proposes a novel approach to automatically segmenting visual speech from facial data using Hidden Markov Models (HMMs) trained solely on visual features. This method aims to capture the unique visual cues of speech articulation, including timing differences and coarticulation, which are crucial for advanced speech animation and understanding.

What Problem Does This Paper Address?

Creating realistic speech animation requires accurately segmenting and labeling speech data. Traditional methods often rely heavily on acoustic information. However, visual speech has its own unique characteristics, such as the timing of facial movements relative to sound production (asynchrony) and how the articulation of one sound influences neighboring sounds (coarticulation). These visual cues are vital for natural-sounding speech animation, especially in noisy environments or for applications like virtual teaching assistants. The challenge is to effectively capture and utilize these purely visual speech characteristics for segmentation, going beyond what acoustic signals alone can provide.

Methods

The core of this research lies in adapting HMM-based forced alignment, a standard technique in Automatic Speech Recognition (ASR), to work with visual speech data. Instead of training HMMs on acoustic features, the authors train them exclusively on features extracted from facial movements captured by a stereo-vision system.

Here's a breakdown of their approach:

  • Feature Extraction: They explore three types of visual features:
  • Articulatory Features: Direct measurements of lip protrusion, lip opening, lip spread, and jaw opening.
  • PCA-based Features: Principal Component Analysis (PCA) applied to markers on the lower face to capture dominant modes of facial deformation related to speech.
  • Combined Features: A fusion of both articulatory and PCA-based features.
  • HMM Training: They train monophone HMMs (models for individual phonemes) using these visual features. This is a standard HMM training process, but with a crucial difference: the input data is purely visual.
  • Forced Alignment: The trained visual HMMs are then used to segment the visual speech data. This process aligns the visual features with a known sequence of phonemes.
  • Context-Dependent Models: To improve accuracy, they further train triphone HMMs (models for phonemes in context) and then use decision tree clustering to create tied-state triphones, leveraging the relationships between phonemes.
  • Evaluation Metric: Segmentation accuracy is evaluated by comparing the predicted visual segments with acoustic segments provided by an ASR engine. A segment is considered correctly recognized if there's any overlap between the predicted visual segment and the actual acoustic segment.

Datasets

The primary dataset used in this study is a custom audio-visual speech corpus.

  • Corpus Details:
    • Duration: Approximately 28 minutes.
  • Content: 319 French sentences covering 35 phonemes (21 consonants, 14 vowels).
  • Acquisition: Captured using a low-cost stereo-vision system with painted markers on the face, synchronized with acoustic speech.
  • Sampling Rate: Data was sub-sampled to 100 Hz and filtered with a low-pass filter at 25 Hz to remove noise while preserving important information.

Key Insights and Ideas

  • Visual Uniqueness: The study demonstrates that HMMs trained solely on visual facial data can effectively capture the unique characteristics of visual speech, including timing differences and coarticulation effects, which are distinct from acoustic cues.
  • Coarticulation Capture: The approach shows promise in automatically labeling visual speech segments based on dominant coarticulatory contexts, which is essential for realistic speech animation.
  • Feature Importance: PCA-based features, which capture overall facial deformation, tend to perform better than purely articulatory features. However, combining both types of features yields the best results.
  • Triphone Advantage: Using context-dependent triphone HMMs significantly improves segmentation performance over monophone models.
  • Deformability Concept: The paper introduces the concept of "deformability" of phonemes – how much a phoneme's articulation can be influenced by its neighbors. This is a key idea for understanding and modeling visual coarticulation.

Key Results

  • Segmentation Performance: The paper presents recognition errors for different feature sets and HMM types (monophone vs. triphone). While specific overall error rates are not explicitly stated as a single headline number, Figure 4 visually compares the performance. The triphone HMMs trained on combined articulatory and PCA features show the lowest recognition errors.
  • Feature Comparison: PCA-based features generally outperform articulatory features. The combination of PCA and articulatory features performs better than either alone.
  • Phoneme Visibility: Recognition errors are lower for "protected" phonemes (those with significant lip-based articulation, like /p/) and those involving labial regions, as expected. Errors are higher for less visually distinct phonemes.
  • Acoustic Baseline: A comparison with HMMs trained on acoustic MFCC features shows a very low recognition error (<1%) for acoustic segmentation, serving as a baseline.

Limitations and Future Work

  • Invisible Phonemes: The segmentation accuracy for "invisible" phonemes (those with less distinct facial articulation) remains a challenge.
  • Tongue Articulation: The current approach does not capture tongue movements, which are crucial for many phonemes. The authors suggest incorporating tongue-related data (e.g., from Electromagnetic Articulography) as a future direction to improve segmentation for invisible phonemes.
  • Deformability Analysis: While the concept of deformability is introduced, further exploration is needed to systematically qualify and label coarticulation effects based on these deformations for improved speech animation.
  • Corpus Size: While the study shows that substantial training can be achieved with a relatively small corpus (28 minutes), larger and more diverse datasets could further enhance robustness.

Summary updated

BibTeX
@inproceedings{musti2010hmm,
  author = {Utpala Musti and Asterios Toutios and Slim Ouni and Vincent Colotte and Brigitte Wrobel-Dautcourt and Marie-Odile Berger},
  title = {{HMM-based automatic visual speech segmentation using facial data}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure