A long, deep and wide artificial neural net for robust speech recognition in unknown noise

Feipeng Li, Phani S. Nidadavolu, Hynek HermanskyInterspeech 2014

TL;DR

This paper introduces the "Long, Deep, and Wide" (LDWNN) artificial neural network, a novel multi-stream architecture designed for robust speech recognition in the face of unpredictable background noise. By breaking down speech into frequency subbands and employing an ensemble of specialized neural networks for each, LDWNN significantly improves recognition accuracy in noisy environments compared to existing methods.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle when the acoustic conditions during testing differ from those encountered during training. Specifically, background noise, which can vary in both its level and spectral characteristics, poses a significant challenge. While training ASR models on a wide range of noisy conditions can improve generalization, it often leads to a degradation in performance on clean speech. Furthermore, it's impractical to anticipate and train for every possible noise scenario. This paper aims to bridge this gap by developing a system that can effectively adapt to and perform well in "unknown noise" – noise that the system hasn't been explicitly trained on.

Methods

The core of the LDWNN system is its multi-stream architecture, inspired by the human auditory system's ability to process sound in parallel across different frequency bands. Here's a breakdown of the key techniques:

  • Frequency Subband Decomposition: The full speech signal is divided into multiple frequency subbands (e.g., 7 subbands for wideband speech). This allows for specialized processing of different frequency ranges.
  • Subband Ensemble Neural Nets: For each subband, instead of a single neural network, an ensemble of neural networks is employed. These ensemble members are trained on speech corrupted by different levels of noise (clean, 20 dB, 10 dB, and 5 dB SNR) using speech-shaped babble noise. This speech-shaped noise is chosen because it mimics the long-term spectral characteristics of speech, providing a more balanced masking effect across frequencies than simple white noise.
  • Neural Network Selector: A crucial component is a neural network selector (a 3-layer MLP) that dynamically chooses the most appropriate subband neural network from the ensemble based on the current acoustic evidence. This allows the system to adapt to varying noise levels within each subband.
  • "Long, Deep, and Wide" Design:
    • Long: The system utilizes relatively long spans of speech signal (300 ms for initial feature extraction, and a concatenated context of 500 ms for final estimation) to capture temporal dynamics.
  • Deep: It employs a multi-stage processing pipeline: feature extraction, subband classification (via the ensemble net), and finally, fusion of information from all subbands.
  • Wide: The "wide" aspect refers to the decomposition of the full speech band into multiple parallel frequency subbands.
  • Feature Extraction: The paper uses Frequency Domain Linear Prediction (FDLP) features, which are designed to capture both short-term spectral and long-term temporal modulation information.
  • Fusion: Outputs from the subband ensemble nets are fused by another neural network to produce the final phoneme sequence.
  • Decoding: A Viterbi algorithm is used for decoding the phoneme sequence.

Datasets

  • TIMIT Speech Corpus: This is the primary dataset used for both training and evaluation. The paper mentions using 3696 sentences for training and 1344 for testing.
  • Speech-Shaped Babble Noise: This type of noise was used for training the ensemble neural networks at various Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Ensemble Learning for Noise Adaptation: The idea of using an ensemble of specialized neural networks, each trained for a specific noise condition, and then having a selector choose the best one is a powerful way to achieve robust performance across a range of unknown noises.
  • Speech-Shaped Noise is Crucial: Using speech-shaped babble noise for training, rather than just white noise, is highlighted as important for achieving balanced masking and better generalization across different frequency bands.
  • Multi-Stream Processing Mimics Human Audition: The paper reinforces the idea that breaking down speech processing into parallel frequency streams, similar to how the human auditory system works, is an effective strategy for robust ASR.
  • "Long, Deep, and Wide" as a Design Philosophy: The combination of extended temporal context ("long"), multi-stage processing ("deep"), and parallel frequency processing ("wide") creates a synergistic effect for improved robustness.

Key Results

  • Superior Generalization: The LDWNN system significantly outperforms single neural networks (trained in clean or mixed conditions) across various frequency subbands, especially in noisy test conditions.
  • Robustness to Unknown Noise: In experiments with various types of unknown noise (subway, babble, factory, restaurant, street, car) at different SNRs, the LDWNN system consistently achieved lower Phoneme Error Rates (PER) compared to baseline single-stream and multi-stream systems.
  • Significant PER Reduction: The LDWNN system achieved relative PER reductions of approximately 20% and 14% compared to multi-stream systems trained on clean and multi-style conditions, respectively. It also outperformed single-stream baselines by a substantial margin (40%, 30%, and 28% relative reduction for MFCC, PNCC, and FDLP features, respectively).
  • Effective with Narrow-Band Noise: Notably, for a low-frequency narrow-band noise like "car" noise at 0 dB SNR, the LDWNN's performance was very close to that in clean conditions, demonstrating its effectiveness against specific noise types.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, multi-stream systems with ensembles and deep architectures inherently have higher computational demands than simpler single-stream models.
  • Generalization to Truly Unseen Noise Types: The paper focuses on "unknown noise" in terms of level and spectral shape variations of known noise types. Future work could explore generalization to entirely novel noise sources.
  • Context Modeling: The paper mentions using context, but further exploration of more sophisticated context modeling techniques could potentially yield further improvements.
  • Fusion Strategy: While the fusion network is effective, exploring alternative fusion mechanisms could be a direction for future research.

Summary updated

BibTeX
@inproceedings{li2014long,
  author = {Feipeng Li and Phani S. Nidadavolu and Hynek Hermansky},
  title = {{A long, deep and wide artificial neural net for robust speech recognition in unknown noise}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure