InQSS: a speech intelligibility and quality assessment model using a multi-task learning network

Yu‐Wen Chen, Yu Tsao

Abstract

Speech intelligibility assessment models are essential tools for researchers to evaluate and improve speech processing models. In this study, we propose InQSS, a speech intelligibility assessment model that uses both spectrogram and scattering coefficients as input features. In addition, InQSS uses a multi-task learning network in which quality scores can guide the training of the speech intelligibility assessment. The resulting model can predict not only the intelligibility scores but also the quality scores of a speech. The experimental results confirm that the scattering coefficients and quality scores are informative for intelligibility. Moreover, we released TMHINT-QI, which is a Chinese speech dataset that records the quality and intelligibility scores of clean, noisy, and enhanced speech.

Introduction

Evaluating speech processing systems—such as speech enhancement (SE) or automatic speech recognition (ASR)—traditionally relies on "intrusive" metrics like PESQ or STOI. These require a clean reference signal, which is rarely available in real-world applications. Furthermore, these metrics often correlate poorly with actual human listening tests.

While non-intrusive neural models have emerged, they typically focus on either quality or intelligibility in isolation. This approach ignores the inherent relationship between how humans perceive these two attributes. Additionally, most existing subjective assessment datasets are in English, leaving a significant gap for other languages like Chinese. InQSS addresses these issues by leveraging multi-task learning and a novel Chinese dataset.

Methodology

The InQSS (Intelligibility and Quality aSSessment) framework treats speech evaluation as a multi-task learning problem. The core technical innovations include:

  • Feature Fusion: In addition to standard spectrograms, the model incorporates scattering coefficients. These are robust, time-shift-invariant signal features that capture complex temporal dynamics often missed by Fourier-based methods.
  • Multi-Task Architecture: The framework predicts quality and intelligibility scores simultaneously. The primary variant, InQSS-MOSNet, utilizes a CNN-BLSTM architecture that processes spectrograms and scattering coefficients through parallel CNN branches before merging them.
  • Joint Training: By training on both tasks, the model learns a shared representation that exploits the high correlation between human-perceived quality and intelligibility.

Other variants include InQSS-SSL, which fine-tunes a pretrained wav2vec 2.0 model, and InQSS-MOSSSL, an ensemble approach that averages predictions from the MOSNet and SSL variants.

The TMHINT-QI Dataset

A key contribution of this work is the release of TMHINT-QI, the first Chinese speech dataset recording subjective quality (1-5 scale) and intelligibility (0-10 scale) scores. It contains 24,408 samples from 226 participants, covering clean, noisy, and enhanced speech. This dataset provides the necessary ground truth for training and evaluating non-intrusive assessment models in a Chinese context.

Results and Analysis

Perception Synergy

Analysis of the TMHINT-QI dataset confirms a strong correlation between human ratings for quality and intelligibility (Pearson Correlation Coefficient = 0.676). As shown in the correlation analysis, samples with high quality almost always exhibit high intelligibility. InQSS leverages this synergy to improve performance on both tasks through shared representations.

Performance Metrics

  • Intelligibility Prediction: The ensemble InQSS-MOSSSL model achieved a Pearson Correlation Coefficient (PCC) of 0.791 on the TMHINT-QI dataset, significantly outperforming the standard STOI metric (PCC: 0.482).
  • Quality Prediction: On the quality task, InQSS-MOSSSL reached a PCC of 0.804, surpassing single-task baselines and existing models like DNSMOS (PCC: 0.496).
  • Feature Importance: Ablation studies revealed that scattering coefficients are particularly informative for intelligibility prediction, consistently reducing Mean Square Error (MSE) compared to models using spectrograms alone.

Conclusion

InQSS demonstrates that joint multi-task learning and the inclusion of scattering coefficients significantly improve the accuracy of non-intrusive speech assessment. While the model aligns well with human perception on the TMHINT-QI dataset, generalization to out-of-domain data remains a challenge for future work. Improving cross-dataset robustness and expanding the framework to more diverse acoustic conditions are key next steps.

Summary updated

Abstract

Speech intelligibility and quality assessment models are essential tools for researchers to evaluate and improve speech processing models. However, only a few studies have investigated multi-task models for intelligibility and quality assessment due to the limitations of available data. In this study, we released TMHINT-QI, the first Chinese speech dataset that records the quality and intelligibility scores of clean, noisy, and enhanced utterances. Then, we propose InQSS, a non-intrusive multi-task learning framework for intelligibility and quality assessment. We evaluated the InQSS on both the training-from-scratch and the pretrained models. The experimental results confirm the effectiveness of the InQSS framework. In addition, the resulting model can predict not only the intelligibility scores but also the quality scores of a speech signal.
BibTeX
@inproceedings{chen2022inqss,
  author = {Yu‐Wen Chen and Yu Tsao},
  title = {{InQSS: a speech intelligibility and quality assessment model using a multi-task learning network}},
  booktitle = {Interspeech},
  year = {2022},
  doi = {10.21437/interspeech.2022-10153},
}
Figure