TL;DR
This paper introduces a novel approach to audio-visual speech recognition (AVSR) by optimizing modality weights on a per-viseme basis, rather than globally. This fine-grained control significantly improves recognition accuracy, especially in noisy conditions, by allowing the system to dynamically leverage the most informative modality for each speech unit.
What Problem Does This Paper Address?
Audio-visual speech recognition (AVSR) aims to improve speech recognition accuracy by combining information from both auditory (speech sounds) and visual (lip movements) signals. A key challenge in AVSR is effectively fusing these two modalities, especially given that they are often naturally asynchronous – lip movements don't perfectly align with the corresponding sounds due to the physics of speech production. Existing methods often use "stream weights" to balance the influence of audio and visual information, but these weights are typically applied globally across all speech units. This paper addresses the limitation that different speech units (phonemes or visemes) might have varying degrees of reliability and informativeness in different acoustic environments. For instance, lip movements might be more crucial for certain sounds than others, or more important when the audio signal is degraded. The authors hypothesize that optimizing these weights on a per-viseme basis, rather than a single global setting, can lead to better performance.
Methods
The researchers implemented their approach within a Coupled Hidden Markov Model (CHMM)-based audio-visual Russian speech recognizer. CHMMs are well-suited for AVSR as they can model the asynchronous nature of audio and visual streams. The core innovation lies in their viseme-dependent stream significance weights. Instead of using a single weight for the audio stream and another for the visual stream across the entire utterance, they assign individual weights to the audio and visual components for each specific viseme-phoneme pair.
The system uses:
- Feature Extraction: 12-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) for audio and 10-dimensional articulatory visual features derived from lip movements (using PCA and LDA). Both streams are synchronized to a 100 Hz frame rate.
- Model Architecture: A 2-stream Coupled Hidden Markov Model (CHMM) is used, which is transformed into an equivalent 2-stream HMM for recognition. This architecture allows for modeling asynchronous streams.
- Weight Optimization: The key contribution is the optimization of weights ($\gamma_{audio}$, $\gamma_{video}$) for each viseme-phoneme model. These weights are applied as exponents to the observation probabilities, effectively controlling the contribution of each modality to the overall likelihood. They explored both global static weights and these new viseme-dependent weights.
Datasets
The experiments were conducted on a privately recorded audio-visual continuous Russian speech corpus. This corpus consists of phonetically-balanced sentences uttered by 10 native Russian speakers in an office environment. The data was recorded with a digital camcorder (720x576 at 25 fps) and a microphone. For training, 60% of each speaker's utterances were used, and the remaining 3-6 connected digit long utterances were used for testing. The paper notes that no publicly available multimodal Russian speech corpus with segmented recordings existed at the time of their study.
Key Insights and Ideas
- Viseme-level Importance: The most significant insight is that the informativeness and reliability of audio and visual speech signals vary not just with the acoustic environment (SNR) but also with the specific speech unit (viseme).
- Asynchrony Modeling is Crucial: The CHMM-based approach, which inherently models asynchrony, outperforms simpler synchronous models like MSHMMs, especially under noisy conditions.
- Visual Cues Vary in Importance: Certain visemes, particularly those involving significant lip motion (e.g., rounded vowels, labial consonants), have a stronger influence on recognition accuracy, especially in noisy environments.
- Silence is Ambiguous Visually: The silence model (V1) consistently benefits from reduced visual stream weights, likely because lip closure during silence can vary significantly between speakers.
- Noise Amplifies Visual Importance: In noisy conditions, visual features become more critical, and the viseme-dependent weighting strategy effectively leverages this by increasing the visual weight for certain visemes.
Key Results
- Overall Improvement: The proposed viseme-dependent weight optimization achieved a relative reduction in Word Error Rate (WER) of 20% compared to the baseline stream-dependent weights, particularly in low SNR conditions.
- CHMM vs. MSHMM: The CHMM-based system consistently outperformed the MSHMM-based system across various SNRs, showing absolute improvements in WRR of up to 6.8%.
- Optimal Weights: For clean speech (25 dB SNR), optimal global weights were found to be $\gamma_{video} = 0.6$ and $\gamma_{audio} = 1.4$. However, viseme-dependent optimization further improved performance.
- Table 2 Summary: At 5 dB SNR, viseme-dependent weights yielded a Word Recognition Rate (WRR) of 65.0% compared to 62.6% for stream-dependent weights. At 25 dB SNR, the improvement was from 96.6% to 97.3%.
Limitations and Future Work
- Computational Cost: While the CHMM-to-HMM transformation increases the number of states, the paper states it does not reduce decoding speed. However, optimizing viseme-dependent weights might add computational overhead during training or adaptation.
- Corpus Specificity: The study was conducted on a Russian AVSR corpus. Generalizability to other languages would require further validation.
- Future Directions: The authors suggest exploring more dynamic weighting strategies and potentially extending this viseme-dependent approach to other AVSR architectures. They also mention the potential for using higher frame rate video cameras in future research.
Summary updated