Some observations on computer lip-reading: moving from the dream to the reality

Helen L. Bear, Gari Owen, Richard W. Harvey, Barry-John TheobaldarXiv 2017 · OpenReview

Abstract

In the quest for greater computer lip-reading performance there are a number of tacit assumptions which are either present in the datasets (high resolution for example) or in the methods (recognition of spoken visual units called visemes for example). Here we review these and other assumptions and show the surprising result that computer lip-reading is not heavily constrained by video resolution, pose, lighting and other practical factors. However, the working assumption that visemes, which are the visual equivalent of phonemes, are the best unit for recognition does need further examination. We conclude that visemes, which were defined over a century ago, are unlikely to be optimal for a modern computer lip-reading system.

Introduction

Computer lip-reading systems are increasingly sought after for applications in high-noise environments, security, and accessibility. Traditionally, these systems mirror the architecture of acoustic speech recognition, substituting audio "phonemes" with "visemes"—groups of phonemes that appear visually identical (e.g., /p/, /b/, and /m/). While recent research has debunked several practical constraints like the necessity for ultra-high resolution or perfect frontal poses, this paper investigates a more fundamental assumption: are visemes actually the optimal unit for visual speech recognition?

Methodology

The study compares parallel audio and visual recognition pipelines to evaluate how each modality processes viseme groupings.

Figure 1: Typical steps in a lip-reading system, from face tracking to recognition.

The visual pipeline utilizes Active Appearance Models (AAMs) for feature extraction, producing two distinct feature sets: shape-only (lip landmarks) and appearance-only (pixel data warped to a mean shape). Recognition was performed using Hidden Markov Models (HMMs) trained with the Hidden Markov Toolkit (HTK). Each of the 15 standard visemes (plus silence) was modeled with 5 states and 5 mixture components. The researchers used the Rosetta Raven dataset, featuring high-quality video of two talkers reciting Edgar Allan Poe’s "The Raven."

Key Insights: The Fragility of Visual Speech

A primary finding is the extreme fragility of lip-reading compared to audio recognition. Visual recognition performance exhibits a "steep drop-off," relying heavily on a small subset of high-performing visemes—primarily silence, certain vowels, and front-of-mouth consonants. While audio recognition draws reliable cues from a broad spectrum of phonetic units, visual models are "starved" for cues and collapse if these few critical visemes are misidentified.

Interestingly, the relative difficulty of recognizing specific visemes remains stable across audio, visual shape, and visual appearance features, suggesting an underlying consistency in the information content of these units.

Results and Discussion

Quantitative analysis revealed that appearance-based features consistently outperform shape-based features. For instance, Talker 2 achieved 52.9% mean accuracy with appearance features versus 32.1% with shape features.

Figure 6: Comparison of recognition performance between Shape and Appearance models for different visemes.

The top-performing visual visemes (excluding silence) were identified as /v04/ (t, d, n, k, g, h, j, ng, y), /v12/ (aa, ao, ah), /v11/ (eh, ae, ey, ay), and /v01/ (p, b, m). The sharp decline in accuracy after these top 5-7 classes highlights the dependency of current systems on a minority of units.

Conclusion

The study suggests that optimizing for mean accuracy may obscure the true performance drivers in lip-reading. Future research should either focus on strengthening these "MVP" visemes or specifically address the failure modes of the worst-performing ones. Ultimately, the century-old concept of the viseme may be suboptimal for modern machine learning, necessitating a re-evaluation of the foundational units of visual speech.

Summary updated

Abstract

In the quest for greater computer lip-reading performance there are a number of tacit assumptions which are either present in the datasets (high resolution for example) or in the methods (recognition of spoken visual units called visemes for example). Here we review these and other assumptions and show the surprising result that computer lip-reading is not heavily constrained by video resolution, pose, lighting and other practical factors. However, the working assumption that visemes, which are the visual equivalent of phonemes, are the best unit for recognition does need further examination. We conclude that visemes, which were defined over a century ago, are unlikely to be optimal for a modern computer lip-reading system.
BibTeX
@misc{bear2017some,
  author = {Helen L. Bear and Gari Owen and Richard W. Harvey and Barry-John Theobald},
  title = {{Some observations on computer lip-reading: moving from the dream to the reality}},
  year = {2017},
  eprint = {1710.01084},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1710.01084},
}
Figure