Specialized Speech Enhancement Model Selection Based on Learned Non-Intrusive Quality Assessment Metric

Ryandhimas E. Zezario, Szu-Wei Fu, Xugang Lu, Hsin-Min Wang, Yu TsaoInterspeech 2019 · DOI

Abstract

Speech enhancement (SE) performance often degrades when general-purpose models encounter environments that differ from their training data. While specialized models (experts) can mitigate this, selecting the optimal expert at runtime typically requires a clean reference signal—a "catch-22" in real-world scenarios. This paper introduces the Specialized Speech Enhancement Model Selection (SSEMS) framework. SSEMS utilizes an ensemble of experts trained on specific gender and noise conditions, paired with Quality-Net, a non-intrusive deep learning model that predicts speech quality (PESQ) without a reference. Experimental results demonstrate that SSEMS consistently outperforms general models and approaches the performance of an "Oracle" selection system.

Introduction

The goal of speech enhancement is to remove noise from a signal to improve its quality and intelligibility. Deep learning has significantly advanced this field, yet "one-size-fits-all" models often struggle with variability in noise types and speaker characteristics. Research indicates that specialized models—trained on specific subsets of data, such as a particular gender or signal-to-noise ratio (SNR) range—can achieve superior results. However, the practical challenge lies in determining which specialized model to use for a given noisy input when the ground-truth clean audio is unknown.

Methodology

The proposed SSEMS framework addresses the selection problem through a two-stage process. First, an ensemble of four specialized Bidirectional Long Short-Term Memory (BLSTM) models is trained. These "experts" are categorized by gender (Male/Female) and SNR level (High/Low).

Second, the framework employs Quality-Net, a non-intrusive quality assessment model. Unlike traditional metrics like PESQ that require a reference signal, Quality-Net is trained to estimate the PESQ score directly from the enhanced audio. During inference, the noisy input is processed by all four experts. Quality-Net evaluates the four outputs and selects the one with the highest predicted quality score.

The SSEMS framework architecture, showing the parallel processing by specialized experts (EM) and the selection by Quality-Net.

Experimental Setup

The system was evaluated using the Wall Street Journal (WSJ) corpus. Training involved 90 different noise types across SNR levels from -10 to 20 dB. Testing was conducted on unseen noise types (car, pink, street, and babble) across seven SNR levels.

Results and Discussion

SSEMS demonstrated significant improvements over a general BLSTM baseline. For instance, in stationary noise at 15 dB SNR, SSEMS achieved a PESQ of 3.29, compared to 3.04 for the baseline.

A key finding is the effectiveness of Quality-Net as a selector compared to traditional methods like Denoising Auto-Encoder (DAE) reconstruction error. As shown in the performance analysis, Quality-Net selection performance closely tracks the "Oracle" selection—the theoretical maximum achieved by using actual PESQ with a clean reference.

PESQ performance comparison between DAE-based selection, Quality-Net selection, and Oracle selection across various SNR levels.

The system's accuracy in picking the best model is highly dependent on the SNR. At 15 dB, selection accuracy reaches approximately 95%, though it drops to near 51% at -10 dB. Furthermore, Quality-Net faces challenges with "babble" noise, where the speech-like characteristics of the background noise can confound the non-intrusive assessment.

Conclusion

The SSEMS framework successfully bridges the gap between specialized model performance and real-world applicability. By leveraging a learned, non-intrusive quality metric, the system can dynamically select the best enhancement strategy. Future work will focus on improving selection accuracy in low-SNR environments and incorporating multiple evaluation metrics into the selection process.

Summary updated

BibTeX
@inproceedings{zezario2019specialized,
  author = {Ryandhimas E. Zezario and Szu-Wei Fu and Xugang Lu and Hsin-Min Wang and Yu Tsao},
  title = {{Specialized Speech Enhancement Model Selection Based on Learned Non-Intrusive Quality Assessment Metric}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2425},
}
Figure