Efficient face model for lip reading

Takeshi SaitohAVSP 2013

TL;DR

This paper investigates which facial features are most effective for lip reading, finding that models focusing on lips, nose, and outline achieve the highest accuracy. Surprisingly, including eyes and eyebrows actually degrades performance, suggesting that lip reading primarily relies on mouth movements and their contextual relation to surrounding facial features.

What Problem Does This Paper Address?

Lip reading, or visual speech recognition, is a valuable technology for noisy environments, communication aids, and situations where audio is undesirable. While many lip reading systems exist, there's been a lack of systematic investigation into which facial features contribute most to accurate recognition. Researchers have built models with varying combinations of facial parts, but direct comparisons of their effectiveness have been difficult due to differing algorithms and datasets. This paper aims to fill this gap by quantitatively analyzing the impact of different facial feature combinations on lip reading accuracy using a consistent experimental setup.

Methods

The core of this research lies in building and evaluating various "face models" for lip reading. The authors employ the Active Appearance Model (AAM), a standard technique for capturing both shape and texture variations in facial images. AAM works by learning a statistical model of facial shape and appearance from training data, allowing it to represent a face with a set of parameters.

The key innovation here is the systematic creation of 20 different face models by varying the combination of facial parts (lips, nose, eyes, eyebrows, outline) and the number of feature points used. These models were then used with a Hidden Markov Model (HMM), a common sequence modeling technique for speech recognition, to perform word recognition. The experiments were designed to isolate the effect of the face model by keeping the recognition algorithm (HMM) and the dataset constant.

Datasets

The study utilizes the CUAVE database. This is a publicly available audio-visual database containing isolated English digits spoken by 36 speakers. For this research, the authors specifically used the "first normal task" of the database, which involves speakers standing still and speaking naturally. The database consists of video frames with a resolution of 720x480 pixels at 29.97 fps, with a green background. The experiments were conducted using data from 30 speakers (16 male, 14 female).

Key Insights and Ideas

  • Lips are essential, but not sufficient: While intuitively obvious, the paper confirms that lip movements are the primary driver of lip reading accuracy.
  • Nose and outline are surprisingly important: The nose and face outline, when combined with lips, significantly boost recognition rates. The authors hypothesize that the nose acts as a stable reference point for lip position, and the face outline's motion correlates with lower lip movement.
  • Eyes and eyebrows are detrimental: Including eyes and eyebrows in the face model actually decreases recognition accuracy. This suggests their movements are largely independent of speech articulation and can introduce noise into the lip reading process.
  • Feature point density matters, but not infinitely: There's an optimal range for the number of feature points on lip contours. Too few or too many points can lead to reduced accuracy, indicating that a moderate, well-chosen set is more effective than simply maximizing the number of points.
  • HMM outperforms DP matching: For the tested models, HMM consistently achieved higher recognition rates than DP matching.

Key Results

The paper reports the following key quantitative findings:

  • The most effective face model (M18), which includes lips, nose, and outline, achieved the highest recognition accuracy of 82.4% using HMM. This model uses 16 points for the external lip contour, 12 for the internal lip contour, 11 for the nose, and 11 for the outline.
  • The base model (M1), representing the whole face, achieved a recognition rate of 69.8% with HMM.
  • Models that included eyes and/or eyebrows (e.g., M7-M10) generally showed lower recognition rates compared to models focusing on lips, nose, and outline.
  • For the best performing models, 12 out of 30 speakers achieved over 90% recognition accuracy.

Limitations and Future Work

Limitations:

  • The study primarily focuses on isolated digit recognition, which may not directly translate to continuous speech.
  • While the CUAVE database is used, the authors acknowledge that the "leave-one-out" method for speaker-dependent recognition might limit generalizability.
  • The paper focuses on the combination of facial parts and feature point density, but further detailed analysis of specific feature point placement within these parts could be explored.

Future Work:

  • Investigating the effectiveness of these face models for continuous speech recognition.
  • Exploring more advanced recognition algorithms beyond HMM.
  • Further refining the number and placement of feature points within the effective facial components (lips, nose, outline).
  • Potentially exploring how these findings can be applied to real-world applications with varying lighting and head pose conditions.

Summary updated

BibTeX
@inproceedings{saitoh2013efficient,
  author = {Takeshi Saitoh},
  title = {{Efficient face model for lip reading}},
  booktitle = {AVSP},
  year = {2013},
}
Figure