Temporal modulation processing of speech signals for noise robust ASR

Hong You, Abeer AlwanInterspeech 2009

TL;DR

This paper introduces a novel frequency-adaptive modulation processing technique to improve the robustness of Automatic Speech Recognition (ASR) systems in noisy environments. By analyzing and selectively denoising speech signals in the modulation domain, the proposed method achieves significant accuracy gains over existing techniques, particularly in challenging noise conditions.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle in the presence of background noise, leading to a significant drop in performance. While traditional approaches focus on spectral features, this paper highlights the crucial role of temporal modulation characteristics of speech for both intelligibility and, importantly, for distinguishing speech from noise. Previous research has shown that low temporal modulation frequencies (below 16 Hz) are vital for speech intelligibility. However, this work delves deeper, investigating how these modulation patterns differ between speech and noise and how additive noise corrupts these patterns. The core challenge addressed is how to effectively leverage these modulation characteristics to build a noise-robust ASR system, going beyond fixed filtering approaches.

Methods

The authors propose a frequency-adaptive modulation processing algorithm. This method operates in the modulation domain, aiming to denoise modulation frequencies that are sensitive to noise while preserving those that are robust. The core idea is to adaptively denoise modulation frequencies below 2 Hz, which are identified as noise-sensitive, by leveraging information from noise-robust modulation frequencies (3-8 Hz) and from a classified "closest robust channel."

Key steps include:

  • Modulation Feature Extraction: Extracting temporal modulation features from the speech signal.
  • Channel Classification: A novel recursive min-max scheme is used to classify acoustic frequency channels into "noise-robust" and "noise-sensitive" based on modulation energy and a "noisiness indicator." This indicator is derived from the difference between modulation frequencies at 0 Hz and 1.5 Hz, aiming to capture smooth energy transitions characteristic of speech.
  • Adaptive Denoising: For noise-sensitive channels, modulation parameters are estimated from the classified noise-robust channel. A denoising equation is then applied to the noisy modulation frequencies, with a threshold to determine if denoising is necessary.
  • Frequency Adaptiveness: Unlike fixed filtering methods (like RASTA), this algorithm adaptively adjusts its processing based on the identified channel characteristics and noise levels.

Datasets

  • TIMIT: Used for training and testing the speech/non-speech modulation feature classifier.
  • Aurora 2 Database: Used for evaluating the performance of the proposed ASR system under various noise conditions. This database includes different noise types (subway, babble, car, exhibition, restaurant, street, airport, train) and Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Speech vs. Noise Modulation Signatures: Speech signals exhibit smoother energy transitions in low modulation frequencies (0-4 Hz) compared to noise, which shows a sharper energy decrease. This difference is a valuable cue for speech/non-speech discrimination.
  • Importance of Low Modulation Frequencies (including DC): Combining DC modulation (0 Hz) with low modulation frequencies is crucial for accurate speech/non-speech classification, suggesting that the static magnitude spectrum's micro-structure is important.
  • Noise Corrupts Modulation Patterns: Additive noise significantly reduces the cross-correlation between low modulation frequencies, necessitating adaptive denoising to maintain consistent modulation patterns for ASR.
  • Frequency Adaptiveness is Key: Adapting the modulation processing based on channel characteristics and noise levels yields better results than fixed filtering approaches.

Key Results

  • Speech/Non-speech Classification: Low-passed modulation features (especially when including DC) performed better than band-passed features for classifying speech from babble noise across different densities.
  • ASR Performance: The proposed frequency-adaptive modulation processing algorithm showed significant improvements in ASR accuracy compared to MFCC and RASTA-MFCC, particularly in noisy conditions (e.g., subway noise).
  • For subway noise on Aurora set A, the proposed method achieved an average accuracy of 68.16%, compared to MFCC (59.77%) and RASTA-MFCC (64.13%).
  • Across various noise types and SNRs on Aurora sets A and B, the proposed method consistently outperformed RASTA-MFCC and, in most cases, approached or exceeded the performance of the more complex ETSI-AFE.
  • Limitations: The proposed algorithm performed less well than RASTA on babble noise, possibly due to babble's smoother modulation energy distribution leading to more classification errors.

Limitations and Future Work

  • Babble Noise Performance: The algorithm's performance degradation on babble noise suggests that further refinement of the channel classification module might be needed for such noise types.
  • Comparison with ETSI-AFE: While the proposed method shows promise, it did not consistently outperform the ETSI-AFE. The authors acknowledge that ETSI-AFE utilizes a more complex set of optimized noise robust techniques.
  • Future Directions: The paper suggests that further investigation into combining spectral and temporal information adaptively, and exploring more sophisticated modulation denoising techniques, could lead to even greater improvements in noise-robust ASR. The authors believe their approach is promising due to its reliance on simpler modulation domain processing compared to more complex front-ends.

Summary updated

BibTeX
@inproceedings{you2009temporal,
  author = {Hong You and Abeer Alwan},
  title = {{Temporal modulation processing of speech signals for noise robust ASR}},
  booktitle = {Interspeech},
  year = {2009},
}

Loading related content...

Figure