Lip-Reading Driven Deep Learning Approach for Speech Enhancement

Ahsan Adeel, Mandar Gogate, Amir Hussain, William M. WhitmerIEEE Trans. Emerg. Top. Comput. Intell 2018 · OpenReview

TL;DR

This paper introduces a novel audio-visual speech enhancement system that combines deep learning-based lip-reading with an enhanced Wiener filter. By leveraging visual cues from lip movements, the system significantly improves speech quality and intelligibility, especially in noisy environments, outperforming traditional audio-only methods.

What Problem Does This Paper Address?

The core problem this paper tackles is speech enhancement, the process of improving the clarity and intelligibility of speech signals that are degraded by noise. While many audio-only methods exist, they often struggle in highly noisy conditions. Human speech perception is naturally multimodal, with visual cues (like lip movements) playing a crucial role in understanding speech, especially when audio is unclear. Existing audio-visual (AV) approaches have shown promise, but this paper aims to bridge the gap between purely deep learning methods and analytical acoustic modeling, proposing a hybrid approach that leverages the strengths of both. Specifically, it addresses the limitations of previous visually-derived Wiener filters (VWF) that relied on less optimal interpolation methods and struggled with generalization.

Methods

The proposed framework operates in two main levels:

  1. Lip-Reading Driven Deep Learning for Feature Estimation:
    • A novel stacked Long-Short-Term Memory (LSTM) network is employed as a lip-reading regression model.
  2. This model takes temporal visual features (lip movements from multiple prior frames) as input to estimate clean audio features. This is a departure from classification-based lip-reading, focusing on regression for speech enhancement.
  3. The LSTM architecture is designed to capture temporal correlations in lip movements, outperforming a conventional Multilayer Perceptron (MLP) in this task.
  4. Enhanced Visually-Derived Wiener Filter (EVWF):
    • The estimated clean audio features from the lip-reading model are then used to derive the clean audio power spectrum.
  5. A new filterbank-domain EVWF is introduced. This filter utilizes an inverse filter-bank (FB) transformation (specifically, a least-squares pseudo-inverse) to approximate the high-dimensional power spectrum from the low-dimensional estimated audio features. This is a key improvement over previous WF methods that used cubic spline interpolation, which struggled with missing spectral values.
  6. Crucially, the EVWF eliminates the need for traditional voice activity detection (VAD) and noise estimation steps, simplifying the process.

Datasets

  • Grid Corpus: Used for training and initial analysis of the lip-reading models (MLP and LSTM). The paper highlights the importance of using aligned sentences to prevent the model from learning silence.
  • ChiME3 Corpus: Used to introduce realistic, dynamic, and commercially relevant noises (cafe, street junction, public transport, pedestrian area) to the Grid corpus. This created a new AV ChiME3 corpus for evaluation.

Key Insights and Ideas

  • Visual Cues are Robust to Noise: The inherent noise immunity of visual speech representations is a significant advantage. Lip movements are unaffected by acoustic noise, making them a reliable source of information for speech enhancement.
  • LSTM Excels at Temporal Visual Information: The stacked LSTM architecture proved superior to MLPs in capturing the temporal dynamics of lip movements across multiple frames, leading to better estimation of clean audio features.
  • Inverse FB Transformation is Key: Replacing cubic spline interpolation with an inverse filter-bank transformation in the Wiener filter significantly improves the estimation of the clean audio power spectrum, especially from low-dimensional features.
  • Hybrid Approach is Powerful: Combining deep learning for feature extraction (lip-reading) with analytical filtering (EVWF) offers a more robust and effective solution than relying solely on one approach.
  • Eliminating VAD/Noise Estimation Simplifies the Pipeline: The EVWF's ability to bypass traditional VAD and noise estimation steps streamlines the speech enhancement process.

Key Results

  • Objective Evaluation (PESQ): The proposed EVWF significantly outperformed conventional Spectral Subtraction (SS) and Log-Minimum Mean-Square Error (LMMSE) methods, especially at low Signal-to-Noise Ratios (SNRs) (-12dB to 0dB). For example, at -12dB SNR, the EVWF showed a substantial improvement in PESQ scores.
  • Subjective Evaluation (MOS): Listening tests confirmed the objective findings. The EVWF achieved significantly higher Mean Opinion Scores (MOS) compared to SS and LMMSE at low SNRs, indicating better perceived speech quality and intelligibility.
  • Lip-Reading Model Performance: The LSTM-based lip-reading model demonstrated better performance than MLP, achieving lower MSE values, particularly when considering multiple prior visual frames. The LSTM performance saturated around 18 prior visual frames.
  • Generalization: The framework was tested on challenging, unseen noisy conditions (e.g., -12dB SNR utterances not included in training), demonstrating good generalization capabilities.

Limitations and Future Work

  • High SNR Performance: While excellent at low SNRs, the AV approach's advantage over audio-only methods diminishes at high SNRs. This is attributed to the nature of spectro-temporally correlated ChiME3 noises.
  • Context-Awareness: The authors suggest future work should focus on developing more optimal, context-aware AV systems that can better handle diverse noisy conditions and integrate visual and noisy audio features more effectively.
  • Generalization and Accuracy: Ongoing work aims to enhance the accuracy and generalization capability of the deep learning lip-reading model.
  • Real-World Data: The team is currently recording data in real conversational settings to create new benchmark resources for the community.
  • Speaker Independence: While a subset of speakers was chosen to ensure speaker independence, further extensive testing with more speakers and datasets is planned.

Summary updated

Abstract

This paper proposes a novel lip-reading driven deep learning framework for speech enhancement. The approach leverages the complementary strengths of both deep learning and analytical acoustic modeling (filtering-based approach) as compared to benchmark approaches that rely only on deep learning. The proposed audio-visual (AV) speech enhancement framework operates at two levels. In the first level, a novel deep learning based lip-reading regression model is employed. In the second level, lip-reading approximated clean-audio features are exploited, using an enhanced, visually-derived Wiener filter (EVWF), for estimating the clean audio power spectrum. Specifically, a stacked long-short-term memory (LSTM) based lip-reading regression model is designed for estimating the clean audio features using only temporal visual features (i.e., lip reading), by considering a range of prior visual frames. For clean speech spectrum estimation, a new filterbank-domain EVWF is formulated, which exploits the estimated speech features. The EVWF is compared with conventional spectral subtraction and log-minimum mean-square error methods using both ideal AV mapping and LSTM driven AV mapping approaches. The potential of the proposed AV speech enhancement framework is evaluated under four different dynamic real-world scenarios [cafe, street junction, public transport, and pedestrian area] at different SNR levels (ranging from low to high SNRs) using benchmark grid and ChiME3 corpora. For objective testing, perceptual evaluation of speech quality is used to evaluate the quality of restored speech. For subjective testing, the standard mean-opinion-score method is used with inferential statistics. Comparative simulation results demonstrate significant lip-reading and speech enhancement improvements in terms of both speech quality and speech intelligibility. Ongoing work is aimed at enhancing the accuracy and generalization capability of the deep learning driven lip-reading model, using contextual integration of AV cues, leading to context-aware, autonomous AV speech enhancement.
BibTeX
@article{adeel2018lip,
  author = {Ahsan Adeel and Mandar Gogate and Amir Hussain and William M. Whitmer},
  title = {{Lip-Reading Driven Deep Learning Approach for Speech Enhancement}},
  journal = {IEEE Trans. Emerg. Top. Comput. Intell.},
  year = {2018},
  url = {https://openreview.net/forum?id=N92VdeL0AK},
}
Figure