Abstract: Cochlear implant (CI) users frequently struggle with speech perception in noisy environments, a phenomenon known as the "cocktail party problem." This study explores the synergy between deep learning-based speech enhancement and visual cues to improve speech intelligibility in CI simulations. By employing a Fully Convolutional Neural Network (FCN) trained with an intelligibility-oriented objective function, the researchers demonstrate significant improvements in comprehension, particularly when paired with visual information (lip-reading). The results suggest that integrating advanced denoising algorithms with audiovisual integration strategies can substantially benefit CI users in challenging acoustic conditions.
Introduction
For individuals with cochlear implants, navigating noisy environments is a persistent challenge. While CIs effectively restore hearing by stimulating the auditory nerve, the resulting signal lacks the spectral resolution of natural hearing, often rendering speech unintelligible in the presence of background noise. Traditional speech enhancement (SE) techniques focus on improving the Signal-to-Noise Ratio (SNR), but "cleaner" audio does not always translate to improved human understanding. This research investigates how modern deep learning and the natural human capacity for audiovisual integration can be combined to address this gap.
Methods
The study utilized a two-part methodology to simulate and enhance the CI experience for normal-hearing participants:
- CI Simulation (Tone-Vocoding): A four-channel tone-vocoder was used to process speech into discrete frequency bands, mimicking the degraded, "robotic" spectral resolution of a cochlear implant.
- FCN Denoising: A Fully Convolutional Neural Network (FCN) was implemented for speech enhancement. Unlike standard frame-based networks, the FCN processes entire utterances to maintain temporal continuity. Crucially, the model was trained using an intelligibility-oriented objective function that optimized for Short-Time Objective Intelligibility (STOI) rather than just minimizing waveform error.
Key Results and Insights
The experimental results, involving 40 participants listening to Mandarin sentences, highlighted several key findings:
- The "Street Noise" Challenge: The FCN was significantly more effective at improving intelligibility in non-stationary "street" noise than in stationary "engine" noise. This is critical as fluctuating noises are typically much harder for the human brain to filter.
- Visual Cues as the Great Equalizer: Visual information (lip-reading) provided a massive boost across all conditions. In the most difficult street noise condition (1dB), adding video skyrocketed intelligibility scores from 11.00 to 51.90.
- Quantitative Gains: Statistical analysis (ANOVA) confirmed that SNR, Video, and the Denoising Condition all had highly significant effects on performance ($p < 2e-16$). The FCN provided the most "added value" at lower SNRs, where the auditory signal was most degraded.
Discussion and Future Work
The study confirms that the joint effect of denoising and visual cues significantly outperforms either method alone. Qualitative analysis showed that the FCN helps reconstruct the rhythmic "shape" of the original speech signal, while visual cues provide the necessary context for the brain to resolve remaining ambiguities.
However, implementing this in real-world devices presents challenges. FCNs are computationally expensive, requiring significant optimization for battery-powered, ear-level CI processors. Future research will focus on actual CI users (rather than simulations) and the development of "fused" audiovisual models where video input directly assists the AI in the denoising process.
Summary updated