Traditional Multi-modality Speech Recognition (MSR) often struggles in high-noise environments because noisy audio signals can degrade the performance of joint audio-visual models. This paper, presented at WACV 2020 by researchers from Xpeng Motors, introduces VE-MSR, a framework that leverages visual information—specifically lip movements—to physically denoise audio signals through a Visual Speech Enhancement (VE) sub-network before the recognition phase.
What Problem Does This Paper Address?
Speech recognition in real-world "in-the-wild" scenarios is frequently hampered by background noise, such as overlapping voices or environmental sounds. While multi-modal systems attempt to mitigate this by incorporating visual cues (lip reading), they typically process noisy audio and video simultaneously. If the audio is severely corrupted, it can act as a distractor, leading to sub-optimal fusion. The authors propose that the visual stream should first be used to filter the audio, identifying and enhancing the speech components that correspond to the observed lip movements.
Methods
The VE-MSR architecture is divided into two main stages:
- Visual Speech Enhancement (VE) Sub-network: This module extracts spatio-temporal features from the speaker's mouth region-of-interest (ROI) using a Pseudo-3D (P3D) residual network. P3D convolutions efficiently capture temporal dynamics by interleaving 1D temporal and 2D spatial filters. These features are used to generate a mask that is applied to the noisy audio's Short-Time Fourier Transform (STFT) magnitude, resulting in an enhanced magnitude spectrogram.
- Multi-modality Speech Recognition (MSR) Sub-network: The enhanced audio and visual features are processed by an encoder-decoder architecture. A key innovation here is the EleAtt-GRU (Element-wise-Attention Gated Recurrent Unit). Unlike standard attention mechanisms that operate at the frame level, EleAtt-GRU applies attention to individual elements of the input vector, allowing the model to focus on fine-grained features crucial for distinguishing subtle phonetic and visual cues.
Datasets and Evaluation
The model was evaluated on two large-scale benchmarks:
- LRW (Lip Reading in the Wild): A dataset of 500 word classes from BBC broadcasts, used for pre-training the visual front-end.
- LRS3-TED: The largest available dataset for sentence-level visual speech recognition, featuring over 400 hours of TED talks, used for main sequence-to-sequence training.
Key Results
The VE-MSR framework demonstrated superior performance, particularly in low Signal-to-Noise Ratio (SNR) conditions:
- Word Accuracy (LRW): Achieved 84.8%, surpassing the previous state-of-the-art of 83.0%.
- Noisy Environments (LRS3-TED): In a 0dB SNR environment (where noise is as loud as the speech), the standard MSR model had a Word Error Rate (WER) of 44.3%. The proposed VE-MSR slashed this to 28.5%.
- Clean Audio: Even with perfectly clean audio, the multi-modal approach improved the WER from 7.2% (audio only) to 6.8%, proving that "watching" helps even when audio is clear.
Conclusion
By treating visual data as an active filter for audio enhancement rather than just a parallel input, VE-MSR effectively addresses the limitations of traditional multi-modal speech recognition in noisy environments. The combination of P3D for efficient temporal modeling and EleAtt-GRU for fine-grained feature attention provides a robust path forward for high-performance ASR in the wild.
Summary updated