Abstract
For a speech-enhancement algorithm, it is highly desirable to simultaneously improve perceptual quality and recognition rate. Thanks to computational costs and model complexities, it is challenging to train a model that effectively optimizes both metrics at the same time. In this paper, we propose a method for speech enhancement that combines local and global contextual structures information through convolutional-recurrent neural networks that improves perceptual quality. At the same time, we introduce a new constraint on the objective function using a language model/decoder that limits the impact on recognition rate. Based on experiments conducted with real user data, we demonstrate that our new context-augmented machine-learning approach for speech enhancement improves PESQ and WER by an additional 24.5% and 51.3%, respectively, when compared to the best-performing methods in the literature.
Introduction
Speech enhancement traditionally focuses on removing background noise to improve signal clarity. Most models are trained to minimize Mean Squared Error (MSE) between noisy and clean signals. However, a significant "metric discrepancy" exists: improvements in perceptual quality (how good it sounds to humans) often degrade the Word Error Rate (WER) for Automatic Speech Recognition (ASR) systems. Denoising filters can inadvertently "smear" phonetic transitions critical for computer understanding. This research addresses this gap by treating speech quality and recognition accuracy as a joint optimization problem.
Methodology
The proposed approach utilizes a two-pronged architecture to bridge the gap between perceptual quality and recognition accuracy.
The CRNN Architecture
To capture both fine-grained local details and long-term temporal patterns, a Convolutional-Recurrent Neural Network (CRNN) is employed. Three convolutional layers analyze windows of eight neighboring audio frames to extract local signal structures. The resulting features are fed into a Long Short-Term Memory (LSTM) network to track global temporal dependencies across the entire audio sequence.
Language Model Constraint and Curriculum Learning
A key innovation of this method is the integration of a language model (decoder) at the end of the denoising network. During training, the model is penalized if the denoised output is difficult for the language model to transcribe correctly. This constraint forces the denoiser to preserve linguistic features essential for ASR.
To manage the complexity of training raw audio frames alongside high-level word sequences, a curriculum learning strategy is used. The denoiser is first trained independently, followed by a fine-tuning phase where the language model constraint is introduced.
Experimental Setup
The study utilized a dataset of 10,500 clean recordings from Microsoft Windows Cortana Voice Assistant queries, mixed with noise from 25 real-world environments. The setup accounted for various room acoustics and microphone distances, resulting in a Signal-to-Noise Ratio (SNR) range of 0-30 dB. Features were extracted using a 512-point Short-Time Fourier Transform (STFT).
Results and Discussion
The proposed model (CRNN + LM + CL) significantly outperformed existing state-of-the-art models. While traditional denoising often doubles or triples the WER (up to 44.93%), our model achieved a WER of 21.90%, a 51.3% improvement over the best existing methods. Furthermore, it achieved a PESQ score of 2.74, representing a 24.5% improvement in perceptual quality.
Key insights include:
- MSE Insufficiency: Minimizing signal error alone can lead to overfitting on noise while losing speech "shape."
- LM as Regularizer: The language model acts as a phonetic "sanity check," ensuring coherent output.
- Contextual Importance: Capturing global temporal dependencies via LSTMs is vital for significant gains in PESQ.
Conclusion
This work demonstrates that combining local and global context with a linguistic constraint effectively balances speech enhancement and recognition accuracy. Future work will explore multi-channel recordings for spatial denoising and model compression techniques to enable real-time deployment on mobile devices.
Summary updated