A HMM recognition of consonant-vowel syllables from lip contours: the cued speech case

Noureddine Aboutabit, Denis Beautemps, Jeanne Clarke, Laurent BesacierInterspeech 2007

TL;DR

This paper explores using Hidden Markov Models (HMMs) to recognize consonant-vowel (CV) syllables from lip movements, specifically within the context of Cued Speech (CS). By incorporating novel lip parameters like "lip pinching," the researchers achieved an 80.3% recognition rate, demonstrating the potential of lip-reading technology to aid individuals with hearing impairments.

What Problem Does This Paper Address?

For individuals with hearing impairments, lip-reading is a crucial visual aid for understanding speech. However, natural lip movements alone can be ambiguous, especially for certain phonemes that look alike on the lips. Cued Speech (CS) is a manual communication system that supplements lip-reading by using handshapes and positions to disambiguate phonemes. The challenge lies in automatically translating this combined visual information (lips + hand cues) into a phonetic representation. This paper focuses on the lip contour aspect, aiming to improve the recognition of basic speech units – consonant-vowel (CV) syllables – using only lip movements, with the ultimate goal of integrating this with CS hand gestures.

Methods

The core of this research employs Hidden Markov Models (HMMs), a standard statistical modeling technique for sequential data like speech. The authors extract various lip parameters from video recordings of a speaker using Cued Speech. These parameters include:

  • Lip Width (A): The horizontal distance across the lips.
  • Lip Aperture (B): The vertical opening between the lips.
  • Lip Area (S): The total area enclosed by the lip contour.

These parameters are derived from both the inner and outer lip contours. A key innovation in this work is the introduction of two new parameters:

  • Lip Pinching (Bsup and Binf): Measures of how much the upper and lower lips are compressed or pinched inwards.

The researchers experimented with different groupings of consonants into "visemes" (visual phonemes) to optimize HMM performance. They also explored the contribution of the new lip pinching parameters by comparing recognition rates with and without them.

Datasets

The study utilized a dataset collected from a native female French speaker who was certified in Cued Speech. She pronounced 267 sentences, repeated at least twice, while performing Cued Speech. The video data was captured at 50 frames per second, and the audio was digitized at 22,050 Hz. The acoustic signal was automatically labeled using forced alignment with French ASR acoustic models. This process generated temporally synchronized lip parameters and phonetic transcriptions for training and testing the HMMs.

Key Insights and Ideas

  • Lip Pinching Matters: The introduction of lip pinching parameters (Bsup and Binf) significantly improved recognition rates, particularly for consonants that are not primarily articulated at the lips. This suggests that subtle lip compressions carry important phonetic information.
  • Consonant Grouping is Crucial: The way consonants are grouped into visemes has a substantial impact on recognition accuracy. The authors iteratively refined these groupings based on phonetic articulation and compatibility with Cued Speech handshapes.
  • Vowel Context is Key: High and mid-high rounded vowels (like [õ, y, o, ø, u]) pose a greater challenge for lip-reading recognition. This is likely due to co-articulation effects, where these vowels can mask the preceding consonant's lip movements.
  • Complementary Information: The study reinforces the idea that lip movements and Cued Speech hand gestures provide complementary information. While this paper focuses on lips, the ultimate goal is to fuse both modalities for more robust speech recognition.

Key Results

The paper presents results from three experiments:

  • Experiment 1 (Initial Grouping): Achieved a global recognition rate of 65.3% using six lip parameters and an initial consonant grouping.
  • Experiment 2 (Modified Grouping): With an improved consonant grouping, the recognition rate increased to 75.9%.
  • Experiment 3 (Lip Pinching): By adding the lip pinching parameters, the global recognition rate reached 80.3%. This represents an improvement of over 4% compared to Experiment 2. The improvement was particularly notable for the challenging C4 consonant group, seeing a gain of over 10%.

The authors also noted that anterior non-rounded vowels and low/mid-low rounded vowels were generally well recognized (around 87% on average).

Limitations and Future Work

  • Contextual Effects: The current models are trained on CV syllables within sentences. The authors acknowledge that the effect of broader sentence context on recognition scores needs further investigation to optimize parameter selection.
  • Tongue Movement: The paper suggests that incorporating parameters related to internal mouth movements, such as tongue gesture, could further improve recognition scores.
  • Error Analysis: While the lip pinching parameters improved overall accuracy, errors persist, particularly with certain consonant-vowel combinations (e.g., C3/C4 consonants with V1 vowels). Further analysis of these confusions is needed.
  • Fusion with Hand Gestures: This work focuses solely on lip contours. Future research will involve integrating these lip-based recognition results with Cued Speech hand gesture information for a more complete phonetic reconstruction.

Summary updated

BibTeX
@inproceedings{aboutabit2007hmm,
  author = {Noureddine Aboutabit and Denis Beautemps and Jeanne Clarke and Laurent Besacier},
  title = {{A HMM recognition of consonant-vowel syllables from lip contours: the cued speech case}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure