TL;DR
This paper introduces a robust 3D lip-tracking system designed for real-world audio-visual speech recognition, particularly in challenging in-vehicle environments. By employing a novel 3D model-based approach and a comprehensive set of 20 lip shape models, the system achieves speaker and illumination independence, paving the way for more accurate and reliable speech recognition systems.
What Problem Does This Paper Address?
Accurate lip-tracking is crucial for audio-visual speech recognition, as it provides vital visual cues that complement acoustic information. Traditional lip-tracking methods often struggle in real-world scenarios due to variations in speaker head pose, illumination, and spontaneous movements. Existing 2D model-based approaches are particularly susceptible to errors when the speaker's head rotates in 3D space, leading to perspective projection issues. This paper tackles the challenge of developing a lip-tracking system that is robust to these real-world variations, aiming for practical applications like in-vehicle speech recognition.
Methods
The core of this work is a 3D model-based lip-tracking approach. The authors first create a set of 20 basic 3D lip shape models derived from real speech utterances of a reference speaker. These models represent the fundamental shapes of lips during phoneme articulation. To manage the complexity, these models are clustered to reduce their number while retaining essential variations.
For tracking, the system utilizes a head-tracking module to determine the 3D position and orientation of the speaker's head. This information is used to define the Region of Interest (ROI) for lip processing. Within the ROI, image processing techniques are employed to extract approximate lip shapes. A key innovation is the use of the H color space to reduce illumination dependency.
The extracted lip shape data is then matched against the pre-defined 3D lip shape models. A distance function is used to quantify the dissimilarity between the observed lip shape and each model. Crucially, a model transition matrix, derived from the training data, acts as a constraint, ensuring that the selected lip shape model for a given frame is plausible given the previous frame's model. This ensures smooth transitions and prevents abrupt changes in the tracked lip shape.
Datasets
The paper introduces and utilizes the Czech In-Vehicle Audio-Visual Speech Corpus (CIVAVC). This corpus consists of 4 hours and 40 minutes of audio-visual speech recordings of a driver in a car. The recordings were made under various traffic conditions. The CIVAVC corpus was specifically collected to address the challenges of real-world audio-visual speech recognition, including spontaneous head movements. The lip shape models were derived from 33 phonetically balanced sentences from a reference speaker within this corpus, totaling approximately 17,000 frames.
Key Insights and Ideas
- 3D Models for Real-World Robustness: The shift from 2D to 3D lip models is a critical step for handling the inherent 3D nature of head movements in real-world scenarios, significantly improving tracking accuracy.
- Clustering for Efficiency: Reducing the number of lip shape models through clustering is a practical approach to manage computational complexity while still capturing the essential variations in lip articulation.
- Illumination Invariance via H Color Space: Employing the H color space for image preprocessing is a clever way to mitigate the impact of varying lighting conditions, a common problem in real-world video.
- Transition Matrix as a Constraint: Using a model transition matrix to guide the selection of lip shape models adds a temporal constraint, making the tracking more coherent and less prone to erroneous jumps between unrelated shapes.
- Speaker Independence: The design aims for speaker independence, meaning the system should work without requiring specific training for each new speaker, which is a significant advantage for practical deployment.
Key Results
The lip-tracking algorithm was tested on 36 utterances from 6 speakers under varying light conditions and 3D head orientations. The total number of frames analyzed was 4290. The paper reports a recognition error rate of 11% for the outer lip boundary and 21% for the inner lip boundary. The higher error rate for the inner boundary is attributed to its dependence on the accurate detection of the outer boundary. The algorithm demonstrated good performance even with darker frames or frames exhibiting significant head position changes, though solarized parts of the face presented a challenge.
Limitations and Future Work
The authors acknowledge that solarized parts of the face can degrade the quality of the lip information and pose a problem for the algorithm. They also note that while the algorithm is designed to be speaker-independent, initial parameters need to be obtained when a new speaker is introduced.
Future work includes developing an automatic speaker adaptation algorithm to streamline the parameter adjustment process for new speakers. They also plan to enhance the image preprocessing algorithm and to test the combined audio-visual recognition system on the collected speech corpus.
Summary updated