Abstract
Speech intelligibility assessment models are essential tools for researchers to evaluate and improve speech processing models. In this study, we propose InQSS, a speech intelligibility assessment model that uses both spectrogram and scattering coefficients as input features. In addition, InQSS uses a multi-task learning network in which quality scores can guide the training of the speech intelligibility assessment. The resulting model can predict not only the intelligibility scores but also the quality scores of a speech. The experimental results confirm that the scattering coefficients and quality scores are informative for intelligibility. Moreover, we released TMHINT-QI, which is a Chinese speech dataset that records the quality and intelligibility scores of clean, noisy, and enhanced speech.
Introduction
Evaluating speech processing systems—such as speech enhancement (SE) or automatic speech recognition (ASR)—traditionally relies on "intrusive" metrics like PESQ or STOI. These require a clean reference signal, which is rarely available in real-world applications. Furthermore, these metrics often correlate poorly with actual human listening tests.
While non-intrusive neural models have emerged, they typically focus on either quality or intelligibility in isolation. This approach ignores the inherent relationship between how humans perceive these two attributes. Additionally, most existing subjective assessment datasets are in English, leaving a significant gap for other languages like Chinese. InQSS addresses these issues by leveraging multi-task learning and a novel Chinese dataset.
Methodology
The InQSS (Intelligibility and Quality aSSessment) framework treats speech evaluation as a multi-task learning problem. The core technical innovations include:
- Feature Fusion: In addition to standard spectrograms, the model incorporates scattering coefficients. These are robust, time-shift-invariant signal features that capture complex temporal dynamics often missed by Fourier-based methods.
- Multi-Task Architecture: The framework predicts quality and intelligibility scores simultaneously. The primary variant, InQSS-MOSNet, utilizes a CNN-BLSTM architecture that processes spectrograms and scattering coefficients through parallel CNN branches before merging them.
- Joint Training: By training on both tasks, the model learns a shared representation that exploits the high correlation between human-perceived quality and intelligibility.
Other variants include InQSS-SSL, which fine-tunes a pretrained wav2vec 2.0 model, and InQSS-MOSSSL, an ensemble approach that averages predictions from the MOSNet and SSL variants.
The TMHINT-QI Dataset
A key contribution of this work is the release of TMHINT-QI, the first Chinese speech dataset recording subjective quality (1-5 scale) and intelligibility (0-10 scale) scores. It contains 24,408 samples from 226 participants, covering clean, noisy, and enhanced speech. This dataset provides the necessary ground truth for training and evaluating non-intrusive assessment models in a Chinese context.
Results and Analysis
Perception Synergy
Analysis of the TMHINT-QI dataset confirms a strong correlation between human ratings for quality and intelligibility (Pearson Correlation Coefficient = 0.676). As shown in the correlation analysis, samples with high quality almost always exhibit high intelligibility. InQSS leverages this synergy to improve performance on both tasks through shared representations.
Performance Metrics
- Intelligibility Prediction: The ensemble InQSS-MOSSSL model achieved a Pearson Correlation Coefficient (PCC) of 0.791 on the TMHINT-QI dataset, significantly outperforming the standard STOI metric (PCC: 0.482).
- Quality Prediction: On the quality task, InQSS-MOSSSL reached a PCC of 0.804, surpassing single-task baselines and existing models like DNSMOS (PCC: 0.496).
- Feature Importance: Ablation studies revealed that scattering coefficients are particularly informative for intelligibility prediction, consistently reducing Mean Square Error (MSE) compared to models using spectrograms alone.
Conclusion
InQSS demonstrates that joint multi-task learning and the inclusion of scattering coefficients significantly improve the accuracy of non-intrusive speech assessment. While the model aligns well with human perception on the TMHINT-QI dataset, generalization to out-of-domain data remains a challenge for future work. Improving cross-dataset robustness and expanding the framework to more diverse acoustic conditions are key next steps.
Summary updated