Performance Comparison of Intrusive Objective Speech Intelligibility and Quality Metrics for Cochlear Implant Users

João Felipe Santos, Stefano Cosentino, Oldooz Hazrati, Philipos C. Loizou, Tiago H. Falk

Introduction

For individuals using Cochlear Implants (CIs), navigating real-world acoustic environments is a significant challenge. Background noise and reverberation (echoes) can severely degrade speech intelligibility. While engineers continuously develop speech enhancement algorithms to mitigate these effects, testing them typically requires subjective evaluations with CI users. This process is time-consuming, expensive, and difficult to scale.

This study, presented at INTERSPEECH 2012, evaluates whether standard objective speech metrics—mathematical models designed to predict human perception—can reliably substitute for subjective testing in the context of CI users.

Methodology

The researchers evaluated six "intrusive" objective metrics, which compare a degraded audio signal against a clean reference to calculate a score:

  • Quality Estimators: PESQ (Perceptual Evaluation of Speech Quality), oPESQ (optimized for reverberation), KLD (Kullback-Leibler Distance), and FWSSRR (Frequency-Weighted Segmental Signal-to-Reconstruction Ratio).
  • Intelligibility Predictors: NCM (Normalized Covariance Metric) and CSII (Coherence Speech Intelligibility Index).

To better reflect the hearing limitations of CI users, who have a narrower dynamic range than normal-hearing individuals, the researchers also tested modified versions of these metrics. Specifically, they narrowed the internal Signal-to-Noise Ratio (SNR) window of the algorithms (e.g., from the standard 30dB range down to a 10dB range) to emulate CI processing.

Experimental Setup

The study utilized a subjective database involving 11 adult CI users listening to sentences from the IEEE corpus. The audio was processed under 12 distinct environmental conditions:

  • Noise-only: Speech-shaped noise at four SNR levels (-5 to 10 dB).
  • Reverberation-only: Four room echo durations (RT60 from 0.3s to 1.0s).
  • Combined: Scenarios featuring both noise and reverberation, mimicking complex real-world settings like crowded restaurants.

Results and Analysis

The evaluation revealed that while most metrics are effective at predicting the impact of background noise, their performance degrades significantly in the presence of reverberation.

The Impact of Reverberation

Reverberation proved to be a major obstacle for objective metrics. In "noise-plus-reverberation" scenarios, most metrics failed to correlate with human performance. For instance, the industry-standard PESQ metric saw its correlation coefficient ($\rho$) drop to a negligible 0.08 in combined conditions, making it effectively useless for evaluating CI speech enhancement in those environments.

Table 1: Performance of objective metrics across different acoustic conditions. Note the significant drop in correlation for most metrics in the Noise-plus-Reverberation scenario.

NCM: The Most Robust Predictor

The Normalized Covariance Metric (NCM) emerged as the most reliable tool across all tested conditions. NCM achieved high Pearson correlations ($\rho \approx 0.93$ to $0.94$ overall), likely because it focuses on the temporal envelope of the speech signal, which is the primary information transmitted by CI processors.

Figure 1: Relationship between subjective intelligibility scores and the NCM metric. The sigmoid curve represents the mapping used to calculate the sigmoid-weighted correlation.

Tuning for CI Limitations

Modifying the metrics to account for the reduced dynamic range of CI users provided modest improvements for some metrics like FWSSRR. However, these adjustments did not fully resolve the inaccuracies observed in complex reverberant environments, suggesting that simple SNR windowing is not a complete solution for modeling impaired hearing.

Conclusion

The study highlights a critical gap in current objective speech evaluation tools for CI applications. While metrics like NCM and CSII show promise, standard quality metrics like PESQ are effectively useless in complex, real-world acoustic scenarios involving both noise and reverberation.

Future research should focus on developing metrics that explicitly model CI signal processing strategies (such as ACE or CIS) and the unique characteristics of electrical stimulation of the auditory nerve, rather than merely adapting models built for acoustic hearing.

Summary updated

BibTeX
@inproceedings{santos2012performance,
  author = {João Felipe Santos and Stefano Cosentino and Oldooz Hazrati and Philipos C. Loizou and Tiago H. Falk},
  title = {{Performance Comparison of Intrusive Objective Speech Intelligibility and Quality Metrics for Cochlear Implant Users}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-471},
}
Figure