Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model based on BLSTM

Szu-Wei Fu, Yu Tsao, Hsin-Te Hwang, Hsin-Min Wang

TL;DR

Quality-Net is a novel, non-intrusive speech quality assessment model that predicts quality scores (like PESQ) directly from raw spectrograms without needing a "clean" reference signal. By using a bidirectional LSTM (BLSTM) architecture with specialized frame-level constraints, it not only provides an overall utterance score but can also pinpoint specific degraded regions within a recording.

What Problem Does This Paper Address?

In speech processing, measuring quality is essential but difficult. Standard objective metrics like the Perceptual Evaluation of Speech Quality (PESQ) are "intrusive," meaning they require a perfectly clean "golden" reference to compare against the degraded signal. In real-world scenarios—like a mobile phone automatically adjusting noise suppression or an ASR system assessing its own input—that clean reference simply doesn't exist.

While humans can easily judge speech quality without a reference, building an "objective, non-intrusive" model is challenging. Previous attempts often relied on complex, handcrafted features that weren't optimized for the final task, or they functioned as "black boxes" that provided a single score without explaining where the quality dropped.

Methods

The authors propose Quality-Net, an end-to-end model based on a Bidirectional Long Short-Term Memory (BLSTM) network.

Figure 1: Proposed Quality-Net for end-to-end, non-intrusive speech quality assessment.

Key Architectural Innovations:

  1. Frame-Level Inference: The model processes a magnitude spectrogram and predicts a quality score ($q_t$) for every single frame. These frame-level scores are then globally averaged to produce the final utterance-level score ($Q$).
  2. Conditional Frame Constraint: Since training data only provides utterance-level labels, the authors introduced a clever loss function. It forces frame-level scores to be uniform and high for high-quality speech. This "guides" the model to recognize that in a "clean" signal, every frame should be good, whereas in a "noisy" signal, only specific frames might be degraded.
  3. Forget Gate Bias (Fgb) Initialization: Standard BLSTMs are designed to remember long-term context. However, for quality assessment, the authors found that too much context causes "bleeding"—noise in one part of a sentence might unfairly lower the predicted quality of a clean part. By initializing the forget gate bias to a small negative value (e.g., -3), they force the model to focus more on the current frame's local characteristics.

Datasets

The researchers used the TIMIT corpus to build their training and testing sets:

  • Training: 4,620 utterances, further divided into clean, noisy (corrupted by 90 different noise types at 8 SNR levels), and enhanced (processed by a separate BLSTM-based enhancer) sets.
  • Testing: 4,900 utterances featuring 4 unseen noise types (engine, white noise, street, and baby crying) to test the model's ability to generalize.
  • Labels: The model was trained to predict PESQ scores, serving as a proxy for human perception.

Key Insights and Ideas

  • Meaningful Intermediate Representations: By designing the model to average frame-level scores, the authors moved away from the "black box" approach. Quality-Net can actually "show its work" by highlighting which parts of an utterance are most degraded.
  • The Power of Constraints: The conditional constraint (Equation 1 in the paper) is vital. Without it, the variance of frame-level scores in clean speech is high; with it, the model learns a stable definition of "clean," significantly improving overall accuracy.
  • Context Control: The finding that limiting LSTM memory (via Fgb initialization) improves frame-level accuracy is counter-intuitive but logical for this task. It prevents the model from being "distracted" by noise that occurred seconds ago.
Figure 2: Example of (a) clean speech, and (b) its correspond- ing frame-level quality assessment by Quality-Net, with and without frame constraint during training.

Key Results

Quality-Net demonstrates impressive alignment with the "intrusive" PESQ standard:

  • High Correlation: It achieved a Linear Correlation Coefficient (LCC) of 0.90 for noisy speech and 0.84 for enhanced speech.
  • Superiority over Baselines: It outperformed existing two-stage models (like those using subband autoencoders) in terms of Mean Square Error (MSE) and Spearman’s Rank Correlation Coefficient (SRCC).
  • Data Efficiency: The model is surprisingly efficient; it reaches a correlation of 0.7 with just 100 training utterances, though performance saturates around 1,000 utterances.
Figure 4: Scatter plots for speech quality assessment by Quali- ty-Net. (a) noisy and clean speech; (b) enhanced speech.

Limitations and Future Work

  • Metric Target: Currently, the model predicts PESQ. While PESQ is a standard, it doesn't always perfectly match human Mean Opinion Scores (MOS). The authors plan to train future versions directly on human-labeled MOS data.
  • Perceptual Loss: Because Quality-Net is end-to-end and differentiable, a major future direction is using it as a "perceptual loss" function. Instead of training speech enhancers to minimize simple mathematical error (like MSE), they could be trained to directly maximize the Quality-Net score, potentially leading to much more natural-sounding results.

Summary updated

Abstract

Nowadays, most of the objective speech quality assessment tools (e.g., perceptual evaluation of speech quality (PESQ)) are based on the comparison of the degraded/processed speech with its clean counterpart. The need of a "golden" reference considerably restricts the practicality of such assessment tools in real-world scenarios since the clean reference usually cannot be accessed. On the other hand, human beings can readily evaluate the speech quality without any reference (e.g., mean opinion score (MOS) tests), implying the existence of an objective and non-intrusive (no clean reference needed) quality assessment mechanism. In this study, we propose a novel end-to-end, non-intrusive speech quality evaluation model, termed Quality-Net, based on bidirectional long short-term memory. The evaluation of utterance-level quality in Quality-Net is based on the frame-level assessment. Frame constraints and sensible initializations of forget gate biases are applied to learn meaningful frame-level quality assessment from the utterance-level quality label. Experimental results show that Quality-Net can yield high correlation to PESQ (0.9 for the noisy speech and 0.84 for the speech processed by speech enhancement). We believe that Quality-Net has potential to be used in a wide variety of applications of speech signal processing.
BibTeX
@inproceedings{fu2018quality,
  author = {Szu-Wei Fu and Yu Tsao and Hsin-Te Hwang and Hsin-Min Wang},
  title = {{Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model based on BLSTM}},
  booktitle = {Interspeech},
  year = {2018},
  doi = {10.21437/interspeech.2018-1802},
}
Figure