Recent developments in automated lip-reading

Richard Bowden, Stephen Cox, Richard Harvey, Yuxuan Lan, Eng-Jon Ong, Gari Owen, Barry-John TheobaldProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2013 · DOI

TL;DR

This paper tackles the challenge of robust automated lip-reading in unconstrained, real-world scenarios, moving beyond controlled lab settings. They introduce a new, challenging dataset and demonstrate improved techniques for camera angle detection and facial feature tracking, paving the way for more reliable lip-reading systems in practical applications like forensic evidence gathering.

What Problem Does This Paper Address?

Automated lip-reading, or video-to-text (V2T) conversion, holds promise for applications like forensic analysis. However, existing systems often perform well in controlled laboratory conditions but struggle in real-world scenarios characterized by poor video quality, varying camera angles, head pose changes, and unconstrained speech. Human lip-readers also suffer from unreliability, highlighting the difficulty of this task. This paper aims to address the limitations of current lip-reading systems by developing and evaluating them on a new, challenging dataset recorded in realistic outdoor environments, and by identifying critical components for robust performance.

Methods

The researchers explored several key areas:

  • Camera Angle Detection: They investigated two methods for identifying the camera's angle relative to the speaker:
  • Active Appearance Model (AAM) fitting error: This approach uses the error from an AAM tracker to determine the best angle.
  • Local Gradient Orientation (LGO) histogram: This method encodes orientation information within facial regions and proved to be more accurate and robust.
  • Robust Facial Feature Tracking: To handle challenging conditions like large pose and expression variations, they developed a new non-linear tracking method based on regression forests, which significantly outperformed previous linear prediction methods.
  • Visual Word Classification: They introduced sequential pattern hypertrees (SP-Hypertrees) with intervals for classifying visual words. This novel approach allows for sharing of common subsequences at any point within patterns, unlike earlier SP-Tree models that only shared initial subsequences. The inclusion of time intervals between feature sets further improved robustness.
  • Discriminative Training: For Hidden Markov Models (HMMs), they explored Minimum Phone Error (MPE) discriminative training as an alternative to conventional Maximum Likelihood (ML) training, showing improved word accuracy.
  • Language Modeling: They evaluated different language models (word-pair grammar, bigram viseme grammar) to understand their impact on recognition performance.

Datasets

The paper introduces a new, two-part outdoor audio-visual dataset:

  • Part 1: A multi-camera dataset of a single subject reading 120 carrier sentences, with city names embedded. This part was recorded using three Sanyo Xacti FH1 cameras at 0°, 30°, and 45° angles, with interlaced video and unconstrained head pose.
  • Part 2: A video sequence of two subjects engaged in unconstrained conversation for approximately 10 minutes. This part was recorded with a handheld camera, introducing more challenging motion blur and camera shake.

They also mention using the LiLIR dataset for training and testing the camera angle detector, which contains multi-camera, multi-angle recordings. The OuluVS dataset was used to evaluate the SP-Hypertree classification model.

Key Insights and Ideas

  • Real-world lip-reading is hard, but tractable: While speaker-independent lip-reading is demanding due to individual variations, aspects like tracking lips on poor-quality interlaced video from hand-held cameras are surprisingly tractable with the right methods.
  • LGO is superior for camera angle detection: The LGO-based method significantly outperforms AAM fitting error for reliably detecting camera angles, likely due to its robustness to minor tracking inaccuracies.
  • Non-linear tracking is crucial for robustness: The new regression forest-based tracker is far more robust to large pose and expression changes than previous linear methods, preventing catastrophic tracking failures.
  • SP-Hypertrees with intervals offer a significant leap: This novel model, allowing arbitrary subsequence sharing and incorporating temporal information, dramatically improves visual word classification accuracy compared to older SP-Tree models.
  • Discriminative training boosts performance: MPE discriminative training shows a substantial relative increase in word accuracy compared to ML training for HMMs.

Key Results

  • Camera Angle Detection: The LGO-based method achieved high accuracy, with confusion matrices showing over 99% accuracy for 30° and 45° angles, and 96.6% for 60° on the outdoor dataset.
  • Robust Tracking: The new regression forest tracker demonstrated superior performance in handling large pose and expression changes compared to linear prediction methods, as visualized in Figure 7.
  • Visual Word Classification: SP-Hypertrees with intervals achieved an average classification rate of 83.5% on the OuluVS dataset, a 22.7% improvement over the original SP-Tree model.
  • Word-Spotting: The system was able to correctly recall all instances of city names in the continuous video stream of the Outdoor Part 2 dataset.
  • Discriminative Training: MPE training resulted in a 53.6% relative increase in word accuracy when using a word-pair grammar.

Limitations and Future Work

  • Person Dependence: The paper acknowledges that person dependence remains a critical problem for lip-reading systems, which they aim to tackle in future work.
  • Directional Lighting: The authors plan to quantify the system's performance under directional lighting changes, using the same regression forest mechanism.
  • Language Model Impact: While they explored language models, further investigation into their optimal use and integration with other components could be beneficial.
  • H Lattice for Error Modeling: The paper mentions a future step of modeling recognition errors using an "H lattice," which could further constrain the output.

Summary updated

Abstract

Human lip-readers are increasingly being presented as useful in the gathering of forensic evidence but, like all humans, suffer from unreliability. Here we report the results of a long-term study in automatic lip-reading with the objective of converting video-to-text (V2T). The V2T problem is surprising in that some aspects that look tricky, such as real-time tracking of the lips on poor-quality interlaced video from hand-held cameras, but prove to be relatively tractable. Whereas the problem of speaker independent lip-reading is very demanding due to unpredictable variations between people. Here we review the problem of automatic lip-reading for crime fighting and identify the critical parts of the problem.
BibTeX
@inproceedings{bowden2013recent,
  author = {Richard Bowden and Stephen Cox and Richard Harvey and Yuxuan Lan and Eng-Jon Ong and Gari Owen and Barry-John Theobald},
  title = {{Recent developments in automated lip-reading}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2013},
  doi = {10.1117/12.2029464},
}
Figure