Modality Attention for End-to-end Audio-visual Speech Recognition

Pan Zhou, Wenwen Yang, Wei Chen, Yanfeng Wang, Jia Jia

TL;DR

Researchers from Tsinghua University and Sogou Inc. have developed a novel "modality attention" mechanism for audio-visual speech recognition (AVSR). By using an LSTM-based scoring function to dynamically weight audio and visual inputs, their system achieves up to a 36% relative improvement over audio-only models in noisy environments, effectively learning to "trust" visual cues more when the audio signal is degraded.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) has reached impressive accuracy in quiet settings, but performance plummets in noisy environments (the "cocktail party problem"). While humans naturally use visual cues—like lip movements—to supplement hearing, teaching machines to do the same is challenging.

The primary hurdle is fusion strategy. Traditional AVSR systems often use simple feature concatenation, which treats audio and video as equally reliable. However, if the audio is corrupted by heavy noise while the video is clear, a "dumb" concatenation can lead the model to rely on garbage data. Previous state-of-the-art models like "Watch, Listen, Attend and Spell" (WLAS) often become over-reliant on audio and are difficult to train. This paper seeks a way to let the model automatically and dynamically adjust its focus between "listening" and "watching" based on the current quality of each stream.

Methods

The authors build upon the Sequence-to-Sequence (Seq2seq) architecture, specifically the "Listen, Attend and Spell" (LAS) framework. Their key innovation is the Modality Attention Mechanism:

  1. Dual Encoders: One encoder (the "Listener") processes acoustic filter banks, while another (the "Watcher") uses a 10-layer CNN followed by a BLSTM to extract visual features from mouth crops.
  2. Modality Attender: Instead of just concatenating the context vectors from the audio and video encoders, the authors introduce a scoring function ($Z$).
  3. LSTM Scoring: Unlike simple feed-forward networks, they use an LSTM to calculate the importance of each modality. This allows the model to consider the temporal history of the signal to decide which stream is currently more reliable.
  4. Dynamic Weighting: The system produces weights ($\alpha$) for each modality at every decoding step. These weights are used to create a fused representation that is then passed to the decoder to predict characters.

Datasets

The study utilized a substantial dataset of broadcast television news videos:

  • Scale: Approximately 150 hours of data, featuring 100 speakers and over 104,000 utterances.
  • Content: Chinese characters (averaging 22 per utterance).
  • Noise Injection: To test robustness, the researchers manually added white Gaussian noise to the original audio at three levels: 10dB, 5dB, and 0dB (where noise is as loud as the speech).
  • Preprocessing: Face detection (DLib) was used to extract 68 landmarks and crop the mouth region into 64x80 pixel RGB images.

Key Insights and Ideas

  • Context-Level Fusion is More Efficient: By performing modality attention on the context vectors (which are already summarized representations) rather than at the raw frame level, the model is computationally more efficient—reducing complexity from $O(T_a T_v)$ to $O(2T_d)$.
  • Temporal Awareness Matters: Using an LSTM for the modality attention scoring function (rather than a static calculation) is crucial. It allows the model to "remember" that the audio has been noisy for the last few seconds, making its decision to rely on video more stable.
  • Automatic Adaptation: The model successfully learned a "preference" without being explicitly told which signal was noisy. As the Signal-to-Noise Ratio (SNR) dropped, the attention weights for the audio modality decreased while visual weights increased.

Key Results

The proposed MD_ATT (Modality Attention) model consistently outperformed audio-only (LAS) and standard concatenation (WLAS) models:

  • Noise Robustness: At 0dB SNR (heavy noise), the Character Error Rate (CER) for the audio-only model was 18.65%. The MD_ATT model slashed this to 11.93%, representing a 36% relative improvement.
  • Clean Performance: Even in clean conditions, the modality attention model showed a slight edge (6.95% CER vs. 7.08% for audio-only), proving that visual cues provide useful information even when the audio is perfect.
  • Weight Shifting: In clean conditions, the model assigned a weight of ~0.64 to audio. At 0dB noise, this dropped to ~0.60, demonstrating the model's ability to shift its focus to the "Watcher" as the "Listener" became less reliable.

Limitations and Future Work

  • Synthetic Noise: The study primarily used synthetic white Gaussian noise. The authors acknowledge that performance should be evaluated against "real-world" noises (like background chatter or traffic).
  • Modality Scope: While this paper focused on audio and video, the authors suggest this attention mechanism could be generalized to other multimodal tasks with correlated information.
  • Dataset Diversity: Future work will involve testing on larger, more diverse AVSR datasets to ensure the findings hold across different languages and recording conditions.

Summary updated

Abstract

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.
BibTeX
@inproceedings{zhou2019modality,
  author = {Pan Zhou and Wenwen Yang and Wei Chen and Yanfeng Wang and Jia Jia},
  title = {{Modality Attention for End-to-end Audio-visual Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=17KMr15vKZ},
}

Loading related content...

Figure