A Study of Joint Effect on Denoising Techniques and Visual Cues to Improve Speech Intelligibility in Cochlear Implant Simulation

Rung-Yu Tseng, Taowei Wang, Szu-Wei Fu, Chia-Ying Lee, Yu TsaoIEEE Trans. Cogn. Dev. Syst 2021 · DOI

Abstract: Cochlear implant (CI) users frequently struggle with speech perception in noisy environments, a phenomenon known as the "cocktail party problem." This study explores the synergy between deep learning-based speech enhancement and visual cues to improve speech intelligibility in CI simulations. By employing a Fully Convolutional Neural Network (FCN) trained with an intelligibility-oriented objective function, the researchers demonstrate significant improvements in comprehension, particularly when paired with visual information (lip-reading). The results suggest that integrating advanced denoising algorithms with audiovisual integration strategies can substantially benefit CI users in challenging acoustic conditions.

Introduction

For individuals with cochlear implants, navigating noisy environments is a persistent challenge. While CIs effectively restore hearing by stimulating the auditory nerve, the resulting signal lacks the spectral resolution of natural hearing, often rendering speech unintelligible in the presence of background noise. Traditional speech enhancement (SE) techniques focus on improving the Signal-to-Noise Ratio (SNR), but "cleaner" audio does not always translate to improved human understanding. This research investigates how modern deep learning and the natural human capacity for audiovisual integration can be combined to address this gap.

Methods

The study utilized a two-part methodology to simulate and enhance the CI experience for normal-hearing participants:

  1. CI Simulation (Tone-Vocoding): A four-channel tone-vocoder was used to process speech into discrete frequency bands, mimicking the degraded, "robotic" spectral resolution of a cochlear implant.
  2. FCN Denoising: A Fully Convolutional Neural Network (FCN) was implemented for speech enhancement. Unlike standard frame-based networks, the FCN processes entire utterances to maintain temporal continuity. Crucially, the model was trained using an intelligibility-oriented objective function that optimized for Short-Time Objective Intelligibility (STOI) rather than just minimizing waveform error.
The FCN architecture processes noisy input utterances through multiple layers to produce an enhanced output, optimized against a clean reference using an intelligibility-based difference measurement.

Key Results and Insights

The experimental results, involving 40 participants listening to Mandarin sentences, highlighted several key findings:

  • The "Street Noise" Challenge: The FCN was significantly more effective at improving intelligibility in non-stationary "street" noise than in stationary "engine" noise. This is critical as fluctuating noises are typically much harder for the human brain to filter.
  • Visual Cues as the Great Equalizer: Visual information (lip-reading) provided a massive boost across all conditions. In the most difficult street noise condition (1dB), adding video skyrocketed intelligibility scores from 11.00 to 51.90.
  • Quantitative Gains: Statistical analysis (ANOVA) confirmed that SNR, Video, and the Denoising Condition all had highly significant effects on performance ($p < 2e-16$). The FCN provided the most "added value" at lower SNRs, where the auditory signal was most degraded.
Performance metrics across various SNR levels demonstrate the consistent improvement provided by the FCN-enhanced speech (FCN_E, FCN_S) compared to noisy conditions (Noisy_E, Noisy_S).

Discussion and Future Work

The study confirms that the joint effect of denoising and visual cues significantly outperforms either method alone. Qualitative analysis showed that the FCN helps reconstruct the rhythmic "shape" of the original speech signal, while visual cues provide the necessary context for the brain to resolve remaining ambiguities.

However, implementing this in real-world devices presents challenges. FCNs are computationally expensive, requiring significant optimization for battery-powered, ear-level CI processors. Future research will focus on actual CI users (rather than simulations) and the development of "fused" audiovisual models where video input directly assists the AI in the denoising process.

Summary updated

BibTeX
@article{tseng2021study,
  author = {Rung-Yu Tseng and Taowei Wang and Szu-Wei Fu and Chia-Ying Lee and Yu Tsao},
  title = {{A Study of Joint Effect on Denoising Techniques and Visual Cues to Improve Speech Intelligibility in Cochlear Implant Simulation}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2021},
  doi = {10.1109/tcds.2020.3017042},
}
Figure