One-Pass Single-Channel Noisy Speech Recognition Using a Combination of Noisy and Enhanced Features

Masakiyo Fujimoto, Hisashi KawaiInterspeech 2019

TL;DR

This paper tackles the challenge of robust speech recognition in noisy environments, specifically for single-channel, real-time applications like smartphones. They propose a novel acoustic modeling approach that combines noisy and speech-enhanced features, along with a gating mechanism to dynamically select the most useful information, significantly improving accuracy under these constraints.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are increasingly integrated into our daily lives through devices like smartphones and smart speakers. However, their performance often degrades significantly in noisy environments. While speech enhancement techniques can help, they often introduce distortions that negatively impact ASR accuracy, especially in single-channel, real-time processing scenarios where complex multi-pass methods are not feasible. Existing advanced acoustic modeling frameworks also struggle to effectively leverage single-channel enhanced speech. This paper aims to bridge this gap by developing a noise-robust ASR method that works effectively within the strict constraints of one-pass, single-channel processing, without sacrificing accuracy.

Methods

The core of the proposed method lies in a novel acoustic modeling framework that intelligently combines noisy and enhanced speech features.

  • Feature-Level Combination: Instead of relying solely on enhanced speech (which can be distorted), the system uses both the original noisy features and the enhanced features. The idea is that in high-SNR (Signal-to-Noise Ratio) periods, enhanced features are beneficial, while in low-SNR periods, the original noisy features might be less distorted and thus more reliable.
  • Sub-Network-Level Combination: To further enhance the combination, the system uses separate sub-networks for processing the noisy and enhanced features. The outputs of these sub-networks are then concatenated, allowing for more complex, non-linear transformations of each feature type before integration.
  • Gating Mechanism: A crucial component is the introduction of a gating mechanism. This mechanism dynamically learns to select the most appropriate features (noisy or enhanced) for each time frame. It acts like a smart switch, deciding which information is most valuable for the ASR system at any given moment, thereby mitigating the impact of speech distortion.
  • Multi-Speech Enhancement Integration: The authors also highlight that their framework can accommodate multiple speech enhancement methods simultaneously. By processing inputs from various enhancement techniques (like SS, MMSE-STSA, GMM-FE, and DAE), the system can leverage the strengths of each, leading to more robust performance across different noise conditions.

Datasets

The proposed methods were evaluated on two datasets:

  • CHiME4 1-channel track: This dataset is designed for noisy speech recognition tasks, featuring recordings from a tablet device in various real-world noisy environments (e.g., public transport, cafeteria). It includes both real and simulated noisy utterances.
  • Corpus of Spontaneous Japanese (CSJ): This is a large-scale corpus of academic lectures in Japanese. To simulate noisy conditions, the authors artificially added four types of noise (airport lobby, exhibition, shopping mall, train station) at varying SNR levels (0-10 dB) to the clean recordings.

Key Insights and Ideas

  • Noisy Features Still Matter: The paper's central insight is that simply enhancing speech isn't always the best strategy. Retaining and intelligently combining the original noisy features with enhanced ones can lead to better ASR performance, especially in challenging low-SNR scenarios.
  • Dynamic Feature Selection is Key: The gating mechanism is a critical innovation. It allows the model to adaptively choose the most reliable feature representation for each moment in the speech signal, effectively overcoming the limitations of fixed enhancement strategies.
  • Ensemble of Enhancement Methods: Combining multiple speech enhancement techniques within the ASR framework offers a significant advantage, suggesting that a "one-size-fits-all" enhancement approach is suboptimal.
  • Constraints Drive Innovation: The paper successfully demonstrates that high-performance noise-robust ASR is achievable even under strict one-pass, single-channel processing constraints, which are common in mobile applications.

Key Results

The proposed method demonstrated significant improvements in Word Error Rate (WER) on both the CHiME4 and CSJ datasets, particularly under noisy conditions.

  • CHiME4: The sub-network-level combination with a gating mechanism, especially when integrating multiple speech enhancement methods (referred to as "ALL"), showed substantial improvements over baseline methods and even individual speech enhancement techniques. For instance, the "ALL" configuration achieved a WER of 11.5% on the CHiME4 Eval set (real data), compared to a baseline of 22.6%.
  • CSJ: Similar positive trends were observed on the CSJ dataset. The "ALL" configuration with the proposed sub-network-level combination and gating mechanism achieved the lowest WERs across both closed and open domains, outperforming conventional methods. For example, it achieved a WER of 11.8% on the CSJ open domain (E02), compared to a baseline of 14.1%.

Limitations and Future Work

  • Limitations: The authors acknowledge that while their method improves ASR accuracy, speech distortion from enhancement can still lead to higher WERs in some cases. The effectiveness of the "ALL" configuration relies on the quality of the individual speech enhancement methods.
  • Future Work: The authors plan to explore more advanced network architectures like Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Convolutional LSTM (CLSTM) for acoustic modeling, as these are better suited for analyzing spatiotemporal information. They also aim to improve individual speech enhancement methods and reduce model complexity by introducing bottleneck structures in sub-networks.

Summary updated

BibTeX
@inproceedings{fujimoto2019one,
  author = {Masakiyo Fujimoto and Hisashi Kawai},
  title = {{One-Pass Single-Channel Noisy Speech Recognition Using a Combination of Noisy and Enhanced Features}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure