Speech estimation in non-stationary noise environments using timing structures between mouth movements and sound signals

Hiroaki Kawashima, Yu Horii, Takashi MatsuyamaInterspeech 2010

TL;DR

This paper introduces a novel method for estimating clean speech signals from noisy audio by leveraging visual cues from mouth movements. By modeling the temporal relationships between lip movements and speech sounds, the system generates multiple clean speech candidates, which are then used to improve noise reduction and speech estimation, especially in challenging non-stationary noise environments.

What Problem Does This Paper Address?

Robust speech recognition in noisy environments is a long-standing challenge. While audio-visual speech recognition (AVSR) methods integrate visual information to improve performance, they often struggle when the signal-to-noise ratio (SNR) of the audio signal becomes very low. Existing AVSR approaches tend to down-weight or ignore audio information in such scenarios, relying more heavily on visual cues. This paper aims to bridge this gap by proposing a method that actively uses visual information to recover clean speech signals from noisy audio, rather than just supplementing audio-based recognition. The core problem is how to effectively extract clean speech information when the audio signal is heavily corrupted by non-stationary noise.

Methods

The proposed method consists of three main phases: learning, candidate generation, and noise compensation.

  • Learning Phase: The authors train Hybrid Dynamical Systems (HDSs) for both speech and visual (mouth movement) features. HDSs are a combination of discrete-event systems (like HMMs) and dynamical systems, allowing them to model complex temporal dynamics. Crucially, they also learn a "cross-media timing-structure model" that captures the temporal relationships and co-occurrence patterns between overlapping speech and visual feature intervals. This model essentially learns how lip movements typically correspond to speech sounds over time, including potential temporal lags.
  • Candidate Generation Phase: Given a noisy audio input and corresponding visual data, the system first uses the trained HDS for visual features to segment the mouth movement sequence into intervals. Then, using the learned timing-structure model, it generates multiple possible interval sequences for speech. Finally, it uses the trained HDS for speech features to generate multiple "clean speech candidates" from these speech interval sequences. This is a key departure from standard AVSR, as it's actively generating speech signals from visual input.
  • Noise Compensation Phase: In this phase, the generated clean speech candidates are combined with the observed noisy audio signal. The authors employ a particle filtering technique, a robust method for tracking non-stationary signals. They model the noisy audio as a combination of clean speech, noise, and observation error. By using the candidates to inform a dynamically changing Gaussian Mixture Model (GMM) for the clean speech component (instead of a static GMM), the particle filter can more effectively estimate the clean speech signal even under low SNR conditions.

Datasets

  • Training and Evaluation Data: The experiments used speech data from a single male subject. This included isolated sounds of the five vowels (/a/, /i/, /u/, /e/, /o/) captured simultaneously with facial images.
  • Noise: Non-stationary noise from the JEITA noise database was used, specifically noise captured in a factory environment including sounds of industrial machines like an air wrench.
  • Audio Specifications: Quantization size of 16-bit and sampling rate of 48 kHz, downsampled to 16 kHz.
  • Image Specifications: Resolution of 640x480 and frame rate of 60 fps.

The paper does not introduce a new benchmark dataset but rather uses existing audio and visual data with added synthetic non-stationary noise for evaluation.

Key Insights and Ideas

  • Visual-to-Audio Generation for Noise Robustness: The core idea is to use visual information not just to aid recognition but to generate a cleaner audio signal from scratch when the original audio is too noisy. This is a powerful paradigm shift for AVSR in low-SNR scenarios.
  • Exploiting Temporal Structures: The success hinges on accurately modeling the timing relationships between mouth movements and speech. This "cross-media timing-structure model" is crucial for generating plausible speech candidates from visual cues.
  • Dynamic GMM for Noise Estimation: Using the generated speech candidates to dynamically update the GMM for clean speech within a particle filter is a clever way to adapt to changing noise conditions and improve estimation accuracy, especially compared to static GMMs.
  • Candidate Ranking and Selection: The ability to generate multiple candidates and rank them by likelihood allows the system to select the most probable clean speech signal, effectively filtering out noise and visual-to-audio mismatches.

Key Results

The paper demonstrates that their proposed method significantly outperforms existing methods (spectral subtraction and particle filtering with static GMM) in estimating clean speech feature sequences under non-stationary noise.

  • Quantitative Improvement: Figure 5 shows that the proposed method achieves much lower error norms between original and estimated speech feature sequences across a range of SNRs, particularly at lower SNRs (closer to 0 dB). For example, at an SNR of 2 dB, the proposed method's error norm is substantially lower than the other two methods.
  • Qualitative Improvement: Figure 4 shows an example of a generated FBANK sequence. The top-ranked candidate (candidate #1) is visually much more similar to the original clean speech ground truth than a lower-ranked candidate (candidate #26), indicating the effectiveness of the candidate generation and ranking process.

Limitations and Future Work

  • Limitations:
    • The evaluation was performed on a single speaker, limiting generalizability.
  • The method assumes no occlusions of the mouth regions in captured images.
  • The current setup uses a single camera and microphone.
  • Future Work:
    • Conduct a comparative evaluation with existing AVSR methods using a large speech corpus.
  • Perform a detailed analysis of the proposed method's characteristics.
  • Extend the method to handle multiple speakers, occlusions, and more complex audio setups (e.g., microphone arrays).

Summary updated

BibTeX
@inproceedings{kawashima2010speech,
  author = {Hiroaki Kawashima and Yu Horii and Takashi Matsuyama},
  title = {{Speech estimation in non-stationary noise environments using timing structures between mouth movements and sound signals}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure