A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement

Ke Tan, DeLiang WangInterspeech 2018

TL;DR

This paper introduces a novel Convolutional Recurrent Neural Network (CRN) for real-time speech enhancement. By combining convolutional encoder-decoder structures with LSTMs, their CRN achieves superior speech intelligibility and quality compared to existing methods, while also being more parameter-efficient and suitable for low-latency applications.

What Problem Does This Paper Address?

Speech enhancement, the process of cleaning up noisy audio, is crucial for many applications like hearing aids and automatic speech recognition. A key challenge is achieving this enhancement in real-time, meaning with minimal delay. Existing methods often struggle to balance performance with the strict latency requirements of real-time systems. Furthermore, robust speech enhancement systems need to generalize well to unseen noise types and speakers, which is difficult for models that rely on limited temporal context.

Methods

The core of this paper is the proposed Convolutional Recurrent Neural Network (CRN). It leverages a convolutional encoder-decoder (CED) architecture, inspired by image segmentation tasks, to extract features from the noisy speech's magnitude spectrogram. This CED is designed with causal convolutions, ensuring that the output at any given time step only depends on past and present inputs, thus enabling real-time processing.

To capture longer-term temporal dependencies, which are important for speaker generalization, two stacked Long Short-Term Memory (LSTM) layers are integrated between the encoder and decoder. The frequency and depth dimensions of the encoder's output are flattened to create sequences that are fed into the LSTMs. The output of the LSTMs is then reshaped to match the decoder's input. This hybrid approach combines the local feature extraction power of CNNs with the sequential modeling capabilities of LSTMs.

Datasets

  • Training: The models are trained on the WSJ SI-84 dataset, which includes 7138 utterances from 83 speakers. To ensure noise independence, 10,000 noise samples from a sound effect library were used, resulting in approximately 126 hours of training data. Mixtures were created using 320,000 utterances with random noise and signal-to-noise ratios (SNRs) ranging from -5 dB to 0 dB.
  • Testing: For evaluation, two challenging noises, babble and cafeteria, were used. Test sets were created for both trained and untrained speakers. Specifically, 150 mixtures were generated for each condition (trained/untrained speakers, babble/cafeteria noise) at SNRs of -5 dB and -2 dB.

Key Insights and Ideas

  • Causality is Key for Real-Time: The paper emphasizes that using causal convolutions is fundamental for achieving real-time speech enhancement, as it strictly prevents the model from looking into the future.
  • CNNs for Feature Extraction, LSTMs for Temporal Context: The CRN effectively combines the strengths of both architectures. CNNs excel at extracting local spectral patterns, while LSTMs are crucial for modeling the temporal dynamics of speech, which aids in speaker generalization.
  • Parameter Efficiency Matters: The CRN demonstrates significantly fewer trainable parameters compared to LSTM baselines, making it more efficient for training and potentially deployment. This is attributed to the weight sharing inherent in convolutional layers.
  • Skip Connections Aid Information Flow: The use of skip connections between encoder and decoder layers helps in preserving fine-grained information and improving gradient flow, leading to better performance.

Key Results

  • Superior Performance: The proposed CRN consistently outperformed two LSTM baselines (one using a temporal window and one without) in terms of both STOI (Speech Intelligibility) and PESQ (Perceptual Evaluation of Speech Quality) scores.
  • Significant Improvements: At an SNR of -5 dB, the CRN achieved approximately 2% higher STOI and 0.1 higher PESQ scores compared to the LSTM models.
  • Excellent Generalization: The CRN showed strong generalization capabilities to untrained speakers, achieving a 18.56% STOI improvement and 0.55 PESQ improvement over unprocessed mixtures in the most challenging test scenario.
  • Faster Convergence: The CRN converged faster and achieved lower Mean Squared Errors (MSEs) during training compared to the LSTM models, as visualized in training curves.
  • Parameter Efficiency: The CRN has significantly fewer trainable parameters (e.g., 17.58 million for CRN vs. 30.22 and 36.81 million for LSTM baselines), making it more efficient.

Limitations and Future Work

  • Limitations: While the CRN demonstrates strong performance, the paper doesn't explicitly detail limitations beyond the inherent challenges of speech enhancement. The focus is on monaural enhancement, which is inherently more difficult than multi-channel approaches.
  • Future Work: The authors suggest that the proposed CRN is a strong candidate for real-world applications that require online operation, single-channel processing, and noise/speaker independence. Further exploration into optimizing the architecture for even lower latency or higher robustness could be future directions.

Summary updated

BibTeX
@inproceedings{tan2018convolutional,
  author = {Ke Tan and DeLiang Wang},
  title = {{A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure