The speaker-independent lipreading play-off; a survey of lipreading machines

Jake Burton, David Frank, Mahdi Saleh, Nassir Navab, Helen L. Bear

Introduction

Machine Lipreading (MLR) has seen significant advancements, yet many reported high-accuracy results are "speaker-dependent," meaning models are tested on individuals they have already seen during training. This creates a misleading benchmark for real-world applications where systems must generalize to unseen speakers with unique mouth shapes and speech patterns. This paper addresses this gap by conducting a systematic "play-off" of various MLR architectures under a strict speaker-independent testing regime using the TCD-TIMIT dataset.

Methodology

The study evaluates a wide array of "pipelines" by combining different feature extraction and classification techniques.

Feature Extraction and Classification

The researchers compared traditional computer vision features—Histogram of Oriented Gradients (HoG), Scale-Invariant Feature Transform (SIFT), Discrete Cosine Transform (DCT), and Active Appearance Models (AAM)—against deep features extracted from a ResNet50 CNN. These features were then processed by various classifiers, including Gaussian Naive Bayes (NB), Random Forests (RF), Support Vector Machines (SVM), and Hidden Markov Models (HMM). Additionally, end-to-end deep learning models like standalone CNNs and LSTMs were tested.

Units of Speech

Performance was measured across two units of speech: phonemes (acoustic units) and visemes (visual units representing mouth shapes). The TCD-TIMIT dataset, consisting of 59 speakers and over 6,000 sentences, was split into a 50-speaker training set and a 9-speaker test set to ensure total speaker independence.

Comparison of accuracy across different feature extraction methods and classifiers for both visemes and phonemes.

Key Insights

The analysis revealed several critical trends in speaker-independent lipreading:

  • Visemes vs. Phonemes: As shown in the comparative plots, visemes consistently outperformed phonemes. This is attributed to the visual ambiguity of certain sounds (e.g., 'p' and 'b' look identical), making the broader viseme categories easier for classifiers to distinguish.
  • The Hybrid Advantage: Contrary to the trend of end-to-end deep learning, the best-performing model was a hybrid: CNN-extracted features paired with a traditional SVM. This suggests that SVMs may handle the specific decision boundaries of visual speech more effectively than standard neural network output layers.
  • HMM Performance: Despite their historical success in audio speech recognition, HMMs performed poorly in this visual, speaker-independent context, likely due to insufficient data variety for generalization across unseen speakers.

Results

The comprehensive "play-off" results are summarized in the heatmaps below, highlighting the performance of every feature-classifier combination.

Heatmaps showing the accuracy of various feature-classifier combinations for phoneme (left) and viseme (right) recognition.

The CNN Image + SVM architecture emerged as the top performer, achieving 69.58% accuracy on viseme classification. For phonemes, the end-to-end CNN reached a peak of 63.15%. Conversely, the combination of SIFT features and HMMs represented the performance floor at just 16.80% for phonemes. While these figures are lower than those typically seen in speaker-dependent studies, they provide a more realistic benchmark for practical lipreading technology.

Conclusion and Future Work

Speaker independence remains a formidable challenge in MLR. The study highlights that the choice of classifier can be as impactful as the feature representation. Future research should focus on domain adaptation to bridge the gap between different speakers' visual spaces and refining temporal models like LSTMs to better capture the grammar of lip movements.

Summary updated

Abstract

Lipreading is a difficult gesture classification task. One problem in computer lipreading is speaker-independence. Speaker-independence means to achieve the same accuracy on test speakers not included in the training set as speakers within the training set. Current literature is limited on speaker-independent lipreading, the few independent test speaker accuracy scores are usually aggregated within dependent test speaker accuracies for an averaged performance. This leads to unclear independent results. Here we undertake a systematic survey of experiments with the TCD-TIMIT dataset using both conventional approaches and deep learning methods to provide a series of wholly speaker-independent benchmarks and show that the best speaker-independent machine scores 69.58% accuracy with CNN features and an SVM classifier. This is less than state of the art speaker-dependent lipreading machines, but greater than previously reported in independence experiments.
BibTeX
@inproceedings{burton2018speaker,
  author = {Jake Burton and David Frank and Mahdi Saleh and Nassir Navab and Helen L. Bear},
  title = {{The speaker-independent lipreading play-off; a survey of lipreading machines}},
  booktitle = {IPAS},
  year = {2018},
  doi = {10.1109/ipas.2018.8708874},
}
Figure