Watch to Listen Clearly: Visual Speech Enhancement Driven Multi-modality Speech Recognition

Bo Xu, Jacob Wang, Cheng Lu, Yandong GuoWACV 2020 · DOI

Traditional Multi-modality Speech Recognition (MSR) often struggles in high-noise environments because noisy audio signals can degrade the performance of joint audio-visual models. This paper, presented at WACV 2020 by researchers from Xpeng Motors, introduces VE-MSR, a framework that leverages visual information—specifically lip movements—to physically denoise audio signals through a Visual Speech Enhancement (VE) sub-network before the recognition phase.

What Problem Does This Paper Address?

Speech recognition in real-world "in-the-wild" scenarios is frequently hampered by background noise, such as overlapping voices or environmental sounds. While multi-modal systems attempt to mitigate this by incorporating visual cues (lip reading), they typically process noisy audio and video simultaneously. If the audio is severely corrupted, it can act as a distractor, leading to sub-optimal fusion. The authors propose that the visual stream should first be used to filter the audio, identifying and enhancing the speech components that correspond to the observed lip movements.

Comparison between (a) standard MSR and (b) the proposed VE-MSR framework which includes an enhancement sub-network.

Methods

The VE-MSR architecture is divided into two main stages:

  1. Visual Speech Enhancement (VE) Sub-network: This module extracts spatio-temporal features from the speaker's mouth region-of-interest (ROI) using a Pseudo-3D (P3D) residual network. P3D convolutions efficiently capture temporal dynamics by interleaving 1D temporal and 2D spatial filters. These features are used to generate a mask that is applied to the noisy audio's Short-Time Fourier Transform (STFT) magnitude, resulting in an enhanced magnitude spectrogram.
  2. Multi-modality Speech Recognition (MSR) Sub-network: The enhanced audio and visual features are processed by an encoder-decoder architecture. A key innovation here is the EleAtt-GRU (Element-wise-Attention Gated Recurrent Unit). Unlike standard attention mechanisms that operate at the frame level, EleAtt-GRU applies attention to individual elements of the input vector, allowing the model to focus on fine-grained features crucial for distinguishing subtle phonetic and visual cues.
Detailed architecture of the Visual Speech Enhancement sub-network and the MSR sub-network components.

Datasets and Evaluation

The model was evaluated on two large-scale benchmarks:

  • LRW (Lip Reading in the Wild): A dataset of 500 word classes from BBC broadcasts, used for pre-training the visual front-end.
  • LRS3-TED: The largest available dataset for sentence-level visual speech recognition, featuring over 400 hours of TED talks, used for main sequence-to-sequence training.

Key Results

The VE-MSR framework demonstrated superior performance, particularly in low Signal-to-Noise Ratio (SNR) conditions:

  • Word Accuracy (LRW): Achieved 84.8%, surpassing the previous state-of-the-art of 83.0%.
  • Noisy Environments (LRS3-TED): In a 0dB SNR environment (where noise is as loud as the speech), the standard MSR model had a Word Error Rate (WER) of 44.3%. The proposed VE-MSR slashed this to 28.5%.
  • Clean Audio: Even with perfectly clean audio, the multi-modal approach improved the WER from 7.2% (audio only) to 6.8%, proving that "watching" helps even when audio is clear.

Conclusion

By treating visual data as an active filter for audio enhancement rather than just a parallel input, VE-MSR effectively addresses the limitations of traditional multi-modal speech recognition in noisy environments. The combination of P3D for efficient temporal modeling and EleAtt-GRU for fine-grained feature attention provides a robust path forward for high-performance ASR in the wild.

Summary updated

BibTeX
@inproceedings{xu2020watch,
  author = {Bo Xu and Jacob Wang and Cheng Lu and Yandong Guo},
  title = {{Watch to Listen Clearly: Visual Speech Enhancement Driven Multi-modality Speech Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093314},
}
Figure