Constrained Convolutional-Recurrent Networks to Improve Speech Quality with Low Impact on Recognition Accuracy

Rasool Fakoor, Xiaodong He, Ivan Tashev, Shuayb Zarar

TL;DR

This paper introduces a speech enhancement model that simultaneously improves audio quality and speech recognition accuracy. By combining a Convolutional-Recurrent Neural Network (CRNN) with a language model constraint and a specialized curriculum learning strategy, the authors achieved a 24.5% improvement in perceptual quality (PESQ) and a massive 51.3% reduction in Word Error Rate (WER) over existing methods.

What Problem Does This Paper Address?

In the world of audio processing, "speech enhancement" usually means removing background noise. Traditionally, these models are trained to minimize the Mean Squared Error (MSE) between the noisy signal and the clean one.

However, there is a "Catch-22" in speech enhancement: making audio sound better to human ears (improving perceptual quality) often makes it harder for computers to understand (increasing the Word Error Rate). This is known as metric discrepancy. A denoising filter might successfully remove a hum but accidentally "smear" the sharp phonetic transitions that an Automatic Speech Recognition (ASR) system needs to identify words. The authors aim to bridge this gap by creating a model that treats speech quality and recognition accuracy as a joint optimization problem.

Methods

The researchers proposed a two-pronged architectural approach:

  1. The CRNN Architecture: To capture both fine-grained local details and long-term patterns, they used a Convolutional-Recurrent Neural Network.
  2. Local Context: Three convolutional layers look at a window of eight neighboring audio frames to understand local signal structures.
  3. Global Context: The output of these convolutions is fed into a Long Short-Term Memory (LSTM) network, which tracks the temporal flow of the entire audio sequence.
  4. The Language Model Constraint: This is the "secret sauce." They attached a language model (decoder) to the end of the denoising network. During training, the model isn't just told to "make the audio clean"; it is also penalized if the resulting clean audio is difficult for the language model to transcribe into the correct text. This forces the denoiser to preserve the linguistic features necessary for ASR.
  5. Curriculum Learning: Training a model to handle both raw audio frames and high-level word sequences simultaneously is incredibly difficult. To solve this, the authors used a curriculum: they first trained the denoiser on its own for several hundred epochs, then introduced the language model constraint later to "fine-tune" the results.

Datasets

The study used a dataset based on real-world user queries to the Microsoft Windows Cortana Voice Assistant.

  • Clean Data: 10,500 studio-level clean recordings.
  • Noisy Data: These clean recordings were mixed with noise collected from 25 different real-world environments.
  • Variability: The researchers accounted for room acoustics and varying distances from the microphone, creating a wide Signal-to-Noise Ratio (SNR) range of 0-30 dB.
  • Features: Audio was processed using a 512-point Short-Time Fourier Transform (STFT), resulting in 256-dimensional feature vectors.

Key Insights and Ideas

  • MSE is Not Enough: The paper demonstrates that simply minimizing signal error (MSE) can lead to overfitting on the noise profile while losing the "shape" of the speech.
  • The LM as a Regularizer: By adding a language model, the authors effectively regularized the denoising process. The LM acts as a "sanity check," ensuring that the denoised output remains phonetically coherent.
  • Context Matters: The comparison between their "NOC" (No Context) model and the "CRNN" model shows that capturing global temporal dependencies via LSTMs is vital for significant jumps in PESQ scores.
  • Training Order is Critical: The authors found that training the denoiser and the language model together from scratch performed poorly. The "Curriculum" approach—mastering the core task before adding the complex constraint—was essential for convergence.

Key Results

The proposed model (CRNN + LM + CL) significantly outperformed both the noisy baseline and several state-of-the-art existing models:

Metric Noisy Data Existing Models (Best) Our Model (Best) Improvement
PESQ (Quality) 2.26 2.20 2.74 +24.5%
WER (Accuracy) 15.4% 44.93% 21.90% +51.3%*

*Note: While the WER of 21.90% is higher than the noisy baseline of 15.4%, it is a massive improvement over other denoising models, which often doubled or tripled the error rate to over 44%. This highlights the paper's success in making denoising "ASR-friendly."

Limitations and Future Work

  • Single-Channel Focus: The research focuses on single-channel recordings. In many modern devices (like smart speakers), multi-microphone arrays are used, which could provide additional spatial data for denoising.
  • Computational Complexity: Combining LSTMs, Convolutions, and a Language Model Decoder creates a heavy model. Future work might look at how to compress this architecture for real-time use on mobile devices.
  • Language Dependency: Since the model uses a specific language model (857-word vocabulary in this study), its performance might vary across different languages or more diverse vocabularies.

Summary updated

Abstract

For a speech-enhancement algorithm, it is highly desirable to simultaneously improve perceptual quality and recognition rate. Thanks to computational costs and model complexities, it is challenging to train a model that effectively optimizes both metrics at the same time. In this paper, we propose a method for speech enhancement that combines local and global contextual structures information through convolutional-recurrent neural networks that improves perceptual quality. At the same time, we introduce a new constraint on the objective function using a language model/decoder that limits the impact on recognition rate. Based on experiments conducted with real user data, we demonstrate that our new context-augmented machine-learning approach for speech enhancement improves PESQ and WER by an additional 24.5% and 51.3%, respectively, when compared to the best-performing methods in the literature.
BibTeX
@inproceedings{fakoor2018constrained,
  author = {Rasool Fakoor and Xiaodong He and Ivan Tashev and Shuayb Zarar},
  title = {{Constrained Convolutional-Recurrent Networks to Improve Speech Quality with Low Impact on Recognition Accuracy}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=n3cln2twVP},
}

Loading related content...

Figure