Introduction
Hearing-impaired individuals rely on hearing aids to navigate noisy environments, yet optimizing these devices requires accurate automated predictions of human speech intelligibility. Traditional "intrusive" predictors—which compare processed audio to a clean reference—often fail to outperform non-intrusive models because they rely on shallow acoustic features. This paper presents the winning approach for the Clarity Prediction Challenge 3 (CPC3), demonstrating how Speech Foundation Models (SFMs) can be effectively adapted for intrusive intelligibility prediction.
The Proposed Architecture
The researchers developed a multi-modal fusion pipeline that leverages the rich phonetic and lexical representations of ASR-based foundation models. The system integrates several key components to bridge the gap between raw audio and human perception.
The core of the model is an SFM backbone (using Canary-1B-flash or Parakeet-tdt) fused with a Multi-Scale Convolutional Neural Network (MSCNN) to preserve fine-grained acoustic details lost during SFM downsampling. A critical innovation is the Reference-Aware Attention mechanism, which performs cross-attention between the degraded ear signals and the clean reference at every stage. Finally, a "best-ear" pooling strategy uses temperature-controlled log-sum-exp pooling to combine binaural scores, mimicking human psychoacoustics where the better-functioning ear often dominates understanding.
Experimental Setup
The model was trained on the CPC3 dataset, comprising over 15,000 signals from 18 enhancement systems and 26 listeners with varying degrees of hearing loss (Mild to Moderately Severe). Evaluation was conducted on a separate set of 7,674 signals. Ground truth labels consisted of human intelligibility scores (0–100%) collected from listeners with hearing impairments.
Key Insights and Results
A major finding of this work is the existence of a "sweet spot" within SFM layers. By performing layer sweeps, the authors discovered that mid-deep layers (specifically layers 10–16) provide the most predictive features for intelligibility.
As shown in Figure 4, these middle layers outperform both early acoustic layers and late abstract linguistic layers. Additionally, the inclusion of a "severity token" — explicitly informing the model of the listener's hearing loss level — yielded significant performance gains.
The final system, dubbed the CPC3 Champion, achieved an RMSE of 24.98 on the evaluation set, securing 1st place in the competition and significantly outperforming the HASPI baseline (RMSE 29.50).
Conclusion
The success of this approach highlights the importance of combining high-level linguistic features from SFMs with explicit reference comparisons and listener-specific metadata. While the model performs exceptionally well on average, future work will focus on addressing "hard" cases where errors remain high, potentially by investigating the impact of specific noise types and reverberation levels.