TL;DR
This paper tackles the challenge of robust automated lip-reading in unconstrained, real-world scenarios, moving beyond controlled lab settings. They introduce a new, challenging dataset and demonstrate improved techniques for camera angle detection and facial feature tracking, paving the way for more reliable lip-reading systems in practical applications like forensic evidence gathering.
What Problem Does This Paper Address?
Automated lip-reading, or video-to-text (V2T) conversion, holds promise for applications like forensic analysis. However, existing systems often perform well in controlled laboratory conditions but struggle in real-world scenarios characterized by poor video quality, varying camera angles, head pose changes, and unconstrained speech. Human lip-readers also suffer from unreliability, highlighting the difficulty of this task. This paper aims to address the limitations of current lip-reading systems by developing and evaluating them on a new, challenging dataset recorded in realistic outdoor environments, and by identifying critical components for robust performance.
Methods
The researchers explored several key areas:
- Camera Angle Detection: They investigated two methods for identifying the camera's angle relative to the speaker:
- Active Appearance Model (AAM) fitting error: This approach uses the error from an AAM tracker to determine the best angle.
- Local Gradient Orientation (LGO) histogram: This method encodes orientation information within facial regions and proved to be more accurate and robust.
- Robust Facial Feature Tracking: To handle challenging conditions like large pose and expression variations, they developed a new non-linear tracking method based on regression forests, which significantly outperformed previous linear prediction methods.
- Visual Word Classification: They introduced sequential pattern hypertrees (SP-Hypertrees) with intervals for classifying visual words. This novel approach allows for sharing of common subsequences at any point within patterns, unlike earlier SP-Tree models that only shared initial subsequences. The inclusion of time intervals between feature sets further improved robustness.
- Discriminative Training: For Hidden Markov Models (HMMs), they explored Minimum Phone Error (MPE) discriminative training as an alternative to conventional Maximum Likelihood (ML) training, showing improved word accuracy.
- Language Modeling: They evaluated different language models (word-pair grammar, bigram viseme grammar) to understand their impact on recognition performance.
Datasets
The paper introduces a new, two-part outdoor audio-visual dataset:
- Part 1: A multi-camera dataset of a single subject reading 120 carrier sentences, with city names embedded. This part was recorded using three Sanyo Xacti FH1 cameras at 0°, 30°, and 45° angles, with interlaced video and unconstrained head pose.
- Part 2: A video sequence of two subjects engaged in unconstrained conversation for approximately 10 minutes. This part was recorded with a handheld camera, introducing more challenging motion blur and camera shake.
They also mention using the LiLIR dataset for training and testing the camera angle detector, which contains multi-camera, multi-angle recordings. The OuluVS dataset was used to evaluate the SP-Hypertree classification model.
Key Insights and Ideas
- Real-world lip-reading is hard, but tractable: While speaker-independent lip-reading is demanding due to individual variations, aspects like tracking lips on poor-quality interlaced video from hand-held cameras are surprisingly tractable with the right methods.
- LGO is superior for camera angle detection: The LGO-based method significantly outperforms AAM fitting error for reliably detecting camera angles, likely due to its robustness to minor tracking inaccuracies.
- Non-linear tracking is crucial for robustness: The new regression forest-based tracker is far more robust to large pose and expression changes than previous linear methods, preventing catastrophic tracking failures.
- SP-Hypertrees with intervals offer a significant leap: This novel model, allowing arbitrary subsequence sharing and incorporating temporal information, dramatically improves visual word classification accuracy compared to older SP-Tree models.
- Discriminative training boosts performance: MPE discriminative training shows a substantial relative increase in word accuracy compared to ML training for HMMs.
Key Results
- Camera Angle Detection: The LGO-based method achieved high accuracy, with confusion matrices showing over 99% accuracy for 30° and 45° angles, and 96.6% for 60° on the outdoor dataset.
- Robust Tracking: The new regression forest tracker demonstrated superior performance in handling large pose and expression changes compared to linear prediction methods, as visualized in Figure 7.
- Visual Word Classification: SP-Hypertrees with intervals achieved an average classification rate of 83.5% on the OuluVS dataset, a 22.7% improvement over the original SP-Tree model.
- Word-Spotting: The system was able to correctly recall all instances of city names in the continuous video stream of the Outdoor Part 2 dataset.
- Discriminative Training: MPE training resulted in a 53.6% relative increase in word accuracy when using a word-pair grammar.
Limitations and Future Work
- Person Dependence: The paper acknowledges that person dependence remains a critical problem for lip-reading systems, which they aim to tackle in future work.
- Directional Lighting: The authors plan to quantify the system's performance under directional lighting changes, using the same regression forest mechanism.
- Language Model Impact: While they explored language models, further investigation into their optimal use and integration with other components could be beneficial.
- H Lattice for Error Modeling: The paper mentions a future step of modeling recognition errors using an "H lattice," which could further constrain the output.
Summary updated