MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids

Ryandhimas E. Zezario, Fei Chen, Chiou‐Shann Fuh, Hsin‐Min Wang, Yu Tsao

Abstract

Improving the user's hearing ability to understand speech in noisy environments is critical to the development of hearing aid (HA) devices. For this, it is important to derive a metric that can fairly predict speech intelligibility for HA users. A straightforward approach is to conduct a subjective listening test and use the test results as an evaluation metric. However, conducting large-scale listening tests is time-consuming and expensive. Therefore, several evaluation metrics were derived as surrogates for subjective listening test results. In this study, we propose a multi-branched speech intelligibility prediction model (MBI-Net), for predicting the subjective intelligibility scores of HA users. MBI-Net consists of two branches of models, with each branch consisting of a hearing loss model, a cross-domain feature extraction module, and a speech intelligibility prediction model, to process speech signals from one channel. The outputs of the two branches are fused through a linear layer to obtain predicted speech intelligibility scores. Experimental results confirm the effectiveness of MBI-Net, which produces higher prediction scores than the baseline system in Track 1 and Track 2 on the Clarity Prediction Challenge 2022 dataset.

Introduction

Developing effective hearing aids requires a reliable way to measure how well a user understands speech in noisy environments. The "gold standard" is subjective listening tests, where humans rate audio samples, but these are prohibitively expensive and slow for large-scale research.

To solve this, researchers use objective metrics. However, most traditional metrics (like STOI or PESQ) are intrusive, meaning they require a perfectly clean version of the audio to compare against the noisy version—a luxury rarely available in real-world scenarios. Furthermore, standard metrics often fail to account for the specific hearing loss profiles of HA users or the binaural (two-ear) nature of hearing. MBI-Net addresses this by providing a non-intrusive (reference-free) prediction model specifically tailored for binaural hearing aid configurations.

Methodology

The authors propose a "divide and conquer" architecture called MBI-Net (Multi-Branched Speech Intelligibility Prediction Model). Its key components include:

  • Dual-Branch Processing: Since HA users often have different hearing capabilities in each ear, the model splits dual-channel audio into two monaural streams. Each stream is processed by its own dedicated branch.
  • Hearing Loss Simulation: Each branch begins with an MSBG model, which modifies the audio signal to simulate the specific hearing loss pattern (audiogram) of the listener.
  • Cross-Domain Feature Extraction: To capture rich acoustic information, the model extracts three types of features: spectral features via STFT, Learnable Filter Banks (LFB), and Self-Supervised Learning (SSL) features using pre-trained models like WavLM.
  • CNN-BLSTM+AT Architecture: The features are fed into a Convolutional Neural Network combined with a Bidirectional Long Short-Term Memory network and a multiplicative attention mechanism to predict frame-level intelligibility.
  • Learned Fusion: Instead of simply averaging the results from both ears, MBI-Net uses a learned linear layer to fuse the outputs, allowing the model to weigh the importance of each channel dynamically.
The overall dual-branch architecture of MBI-Net, showing the split-channel processing and final fusion.

Experimental Setup and Datasets

The model was evaluated using the Clarity Prediction Challenge 2022 dataset, which includes:

  • Subjective Scores: Intelligibility ratings (0–100) from 25 HA users.
  • Audiograms: Pure-tone hearing thresholds for each listener.
  • Track 1 (Closed-set): 4,863 training utterances and 2,421 test utterances.
  • Track 2 (Open-set): 3,580 training utterances and 632 test utterances.

Results and Discussion

MBI-Net demonstrated significant improvements over the challenge baseline across three metrics: Root Mean Square Error (RMSE), Standard Deviation Error (STDERR), and Linear Correlation Coefficient (LCC).

  • Track 1 (Closed-set): The optimized MBI-Net (WavLM+) achieved an LCC of 0.78 and an RMSE of 23.05, compared to the baseline's LCC of 0.62 and RMSE of 28.52.
  • Track 2 (Open-set): On the more difficult open-set data, MBI-Net (WavLM+) maintained a high LCC of 0.75, while the baseline dropped to 0.53.

Qualitatively, scatter plots show that MBI-Net’s predictions align much more closely with the diagonal "true intelligibility" line than the baseline, particularly in the mid-to-high intelligibility ranges. This indicates that the inclusion of WavLM features and the dual-branch architecture effectively captures the nuances of speech intelligibility for hearing-impaired listeners.

Scatter plots comparing the baseline system and MBI-Net (wavLM+) against true subjective intelligibility scores.

Conclusion

MBI-Net represents a significant step forward in non-intrusive speech intelligibility prediction. By leveraging self-supervised speech representations (WavLM) and a binaural-aware architecture, it provides more accurate surrogates for human listening tests. Future work will explore advanced feature engineering and refined neural architectures to better model complex binaural interactions and reduce computational complexity for real-time applications.

Summary updated

BibTeX
@inproceedings{zezario2022mbi,
  author = {Ryandhimas E. Zezario and Fei Chen and Chiou‐Shann Fuh and Hsin‐Min Wang and Yu Tsao},
  title = {{MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids}},
  booktitle = {Interspeech},
  year = {2022},
  doi = {10.21437/interspeech.2022-10838},
}
Figure