Abstract
Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.
Introduction
Automatic Speech Recognition (ASR) has achieved remarkable accuracy in controlled, quiet environments. However, performance often degrades significantly in noisy settings, a challenge known as the "cocktail party problem." While humans naturally integrate visual cues—such as lip movements—to supplement hearing in loud environments, replicating this multisensory integration in machines remains difficult.
The primary challenge lies in the fusion strategy. Traditional AVSR systems often rely on simple feature concatenation, which treats audio and video streams as equally reliable regardless of signal quality. This can lead to poor performance if a corrupted audio stream is given equal weight to a clear visual stream. This paper introduces a dynamic "modality attention" mechanism that allows a model to automatically adjust its focus between "listening" and "watching" based on the real-time reliability of each input.
The Modality Attention Framework
Building upon the Sequence-to-Sequence (Seq2seq) architecture, specifically the "Listen, Attend and Spell" (LAS) framework, the proposed system utilizes a novel Modality Attention Mechanism:
- Dual Encoders: The system employs a "Listener" encoder for acoustic filter banks and a "Watcher" encoder. The Watcher uses a 10-layer CNN followed by a BLSTM to extract visual features from cropped mouth images.
- Modality Attender: Rather than simple concatenation, the model introduces a scoring function ($Z$) to determine the importance of each modality.
- LSTM-Based Scoring: By using an LSTM to calculate modality weights, the system incorporates temporal history. This allows the model to "remember" if a signal has been consistently noisy, leading to more stable weighting decisions.
- Dynamic Weighting: The system generates weights ($\alpha$) for each modality at every decoding step, creating a fused representation optimized for character prediction.
Experimental Setup
The model was evaluated using a large-scale dataset of broadcast television news, comprising approximately 150 hours of data across 100 speakers and 104,000 utterances. To test robustness, white Gaussian noise was added to the original audio at three levels: 10dB, 5dB, and 0dB (where noise intensity equals speech intensity). Visual preprocessing involved face detection and cropping the mouth region into 64x80 pixel RGB images.
Results and Discussion
The proposed MD_ATT (Modality Attention) model consistently outperformed both audio-only (LAS) and standard concatenation (WLAS) baselines across all noise levels.
Noise Robustness and Performance
As shown in the performance data, the MD_ATT model demonstrates significant resilience to noise. At 0dB SNR, the audio-only LAS model recorded a Character Error Rate (CER) of 18.65%. The MD_ATT model reduced this to 11.93%, a 36% relative improvement. Even in clean conditions, the modality attention model maintained a slight advantage (6.95% CER vs. 7.08% for LAS), suggesting that visual cues provide valuable context even when audio is clear.
Adaptive Weight Shifting
A key finding of this research is the model's ability to automatically shift its reliance between modalities. Analysis of the attention weights reveals that as the Signal-to-Noise Ratio (SNR) decreases, the model decreases the weight assigned to the audio stream ($\alpha^a$) and increases the weight for the visual stream ($\alpha^v$).
In clean conditions, the model assigned a weight of approximately 0.641 to audio. At 0dB noise, this weight dropped to 0.607, confirming that the system learned to "trust" the visual encoder more as the auditory signal became less reliable.
Conclusion
This work demonstrates that a dynamic modality attention mechanism significantly enhances the robustness of speech recognition in noisy environments. By performing fusion at the context level and utilizing temporal scoring, the model efficiently integrates audio and visual data. Future research will focus on evaluating the system against real-world noise types and exploring the generalization of this attention mechanism to other multimodal tasks.
Summary updated