Abstract
Speech Intelligibility Prediction (SIP) is a critical component in the evaluation and optimization of hearing aids, telecommunications systems, and speech enhancement algorithms. While intrusive SIP methods—which require a clean reference signal—have reached high levels of accuracy, their utility is limited in real-world scenarios where only the degraded signal is available. This paper presents a non-intrusive SIP model based on a Convolutional Neural Network (CNN) architecture. By mimicking the structure of traditional auditory models and focusing on interpretable spectro-temporal features, the proposed CNN achieves a Root-Mean-Square Error (RMSE) of 17.69 percentage points, rivaling the performance of the intrusive STOI benchmark and significantly outperforming existing non-intrusive alternatives.
Introduction
The ability to automatically predict how well a human listener can understand a speech signal is essential for iterative hardware and software development. Traditionally, this assessment relies on subjective listening tests, which are resource-intensive. Automated SIP algorithms serve as a proxy for these tests. However, the most effective algorithms are "intrusive," requiring access to the original, undistorted speech. In practical applications like hearing aids operating in noisy environments, such a reference is non-existent.
Non-intrusive SIP aims to bridge this gap. Historically, non-intrusive methods have struggled with complex noise and non-linear distortions. This research leverages deep learning to create a robust, non-intrusive predictor that generalizes across different languages and noise conditions.
Problem Definition
The core challenge of non-intrusive SIP is estimating the impact of degradation without knowing the underlying clean signal. As shown in Figure 1, intelligibility is a time-varying property that is only defined during active speech segments. A successful model must distinguish between speech-like modulations and interfering noise.
Proposed Methodology
The authors propose a CNN architecture specifically tailored for auditory processing. The design prioritizes interpretability, ensuring that the network's internal representations align with known psychoacoustic principles.
Preprocessing and Feature Extraction
Audio signals are resampled to 10 kHz, and 15 one-third octave band envelopes are extracted. These envelopes are normalized for mean and variance to ensure the model remains invariant to absolute signal levels, focusing instead on temporal fluctuations.
CNN Architecture
The network structure is designed to be relatively shallow, reflecting the hypothesis that SIP is a simpler task than Automatic Speech Recognition (ASR).
- Convolutional Layer: The model uses $K$ kernels that span all 15 frequency bands with a temporal width of 384 ms. This duration is sufficient to capture phonemic and syllabic transitions.
- Activation and Pooling: A Softplus activation function provides a smooth non-linearity. Global average pooling then collapses the temporal dimension, summarizing the features detected across the entire utterance.
- Dense Layers and Output: Three feed-forward layers process the pooled features, followed by a logistic output layer that maps the results to a 0–100% intelligibility score.
Mitigating Overfitting
To prevent the model from memorizing the limited vocabulary of standard intelligibility datasets, the researchers employed a "speech substitution" technique. Training stimuli were recreated using the large-scale Danish ADFD corpus, ensuring the model learned acoustic features rather than specific words.
Experimental Results
The model was evaluated against several prominent datasets covering various noise types, including stationary noise, modulated noise, and filtered speech.
- Accuracy: The CNN achieved an RMSE of 17.69 pp, a substantial improvement over the non-intrusive SRMR baseline (32.77 pp).
- Benchmark Comparison: Remarkably, the CNN's performance was statistically comparable to the intrusive STOI measure (18.94 pp), despite the latter having the advantage of a clean reference signal.
- Generalization: The model demonstrated strong cross-language generalization, performing accurately on Dutch matrix sentence tests without retraining.
Discussion and Key Insights
A significant finding of this work is the interpretability of the learned kernels. Visualization showed that the network naturally optimized for modulations in the 3–5 Hz range—the same frequencies humans rely on most for speech comprehension. This suggests that the CNN is not merely a "black box" but has learned fundamental properties of speech perception.
While the model excels in many conditions, highly fluctuating noise remains a challenge. Future work may involve expanding the architecture to support binaural signals, mimicking the human ability to use spatial cues for noise cancellation.
Conclusion
This research demonstrates that a carefully designed, non-intrusive CNN can match the performance of intrusive SIP algorithms. By focusing on temporal modulations and utilizing diverse training data, the model provides a viable path for real-time intelligibility assessment in hearing assistive devices and telecommunications.
Summary updated