End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments

Meng-Ping Lin, Enoch Hsin-Ho Huang, Shao-Yi Chien, Yu Tsao

TL;DR

This paper introduces AVSE-ECS, a novel end-to-end system for cochlear implant (CI) sound coding that leverages audio-visual speech enhancement (AVSE) to improve speech intelligibility in noisy environments. By integrating visual cues like lip movements with traditional audio processing, AVSE-ECS significantly boosts speech understanding, achieving a remarkable 7.4666 dB improvement in signal-to-error ratio compared to existing methods.

What Problem Does This Paper Address?

Individuals with severe-to-profound hearing loss often rely on cochlear implants (CIs) to perceive sound. While CIs are highly effective in quiet environments, their performance degrades significantly in the presence of background noise or competing speech. Traditional CI sound coding strategies, like the Advanced Combination Encoder (ACE), struggle to isolate the target speech signal in these challenging conditions. Existing speech enhancement (SE) techniques for CIs have primarily focused on audio-only processing, neglecting the valuable visual information (e.g., lip movements) that humans naturally use to aid comprehension in noisy situations. This paper addresses the critical need for CI systems that can effectively utilize multimodal information to enhance speech perception in noisy environments.

Methods

The core innovation is the development of the AVSE-ECS system, which combines an Audio-Visual Speech Enhancement (AVSE) module with the ElectrodeNet-CS (ECS) model.

  • ECS Model: This is a deep neural network (DNN) designed to replicate the signal processing of the ACE strategy, but in a differentiable manner that allows for end-to-end training. It takes spectral-temporal matrices as input and outputs electrode stimulation patterns.
  • AVSE Module: This module is built upon the NCSN++ model (a U-Net architecture) and incorporates a Temporal Convolutional Network (TCN) as a visual encoder. The TCN processes lip movements extracted from video.
  • Audio-Visual Fusion: A key component is the cross-attention block, which allows the model to fuse audio and visual features. Specifically, it modifies self-attention blocks by using visual embeddings as keys and values for the audio input, enabling the model to focus on relevant visual cues that correspond to the audio.
  • Joint Training: The AVSE and ECS modules are trained together end-to-end. This joint training optimizes two loss functions:
  • Spectrogram Loss ($L_{Spec}$): Measures the mean squared error between the enhanced speech spectrogram and the clean speech spectrogram.
  • Electrodogram Loss ($L_{Elec}$): Measures the mean squared error between the enhanced electrodogram (output of AVSE-ECS) and the clean electrodogram (output of ECS with clean speech).
  • The total loss ($L_{Total}$) is a weighted sum of these two losses ($\alpha L_{Spec} + \beta L_{Elec}$).

Datasets

  • Taiwan Mandarin Hearing in Noise Test (TMHINT): Used for training the audio-only ECS model. It consists of phonemically balanced lists of Mandarin sentences.
  • Taiwan Mandarin Speech with Video (TMSV): This is the primary dataset used for training and evaluating the AVSE-ECS system. It comprises speech and synchronized video recordings of TMHINT sentences from multiple Mandarin speakers. The dataset includes speech mixed with various noise types at different Signal-to-Noise Ratios (SNRs). For evaluation, a separate set of unseen speakers and noise conditions were used to ensure robust generalization.

Key Insights and Ideas

  • Visual Cues are Crucial for Noisy CI Environments: The study strongly suggests that incorporating visual information (lip movements) significantly enhances speech intelligibility for CI users in noisy conditions, outperforming audio-only approaches.
  • End-to-End Learning Bridges the Gap: By enabling joint training of AVSE and ECS, the system can learn to generate more accurate and informative electrode stimulation patterns, directly addressing the limitations of traditional, non-differentiable coding strategies.
  • Cross-Attention for Effective Fusion: The use of cross-attention mechanisms proves effective in aligning and integrating audio and visual information, allowing the model to leverage visual cues that are most relevant to the speech signal.
  • Joint Training Outperforms Pre-training: While pre-training the AVSE module and then concatenating it with ECS yields improvements, the end-to-end joint training approach achieves superior results, indicating that simultaneous optimization of both modules is more beneficial.

Key Results

  • The ECS model alone shows a significant drop in performance in noisy conditions (STOI of 0.4870) compared to clean speech (STOI of 0.7604).
  • The AVSE-ECS system with joint training achieved the highest objective speech intelligibility scores.
  • It improved the Signal-to-Error Ratio (SER) by 7.4666 dB compared to the ACE strategy.
  • Achieved a STOI score of 0.6305 in noisy conditions, a substantial improvement over the ECS model (0.4870) and the audio-only ASE-ECS model (0.5943).
  • The optimal weight for the electrodogram loss ($\beta$) in joint training was found to be 0.5, balancing the benefits of spectrogram and electrodogram refinement.

Limitations and Future Work

  • Single Language and Dataset: The current evaluation relies on a single dataset (TMSV) and language (Mandarin Chinese). While regression tasks are often language-agnostic, further studies with diverse datasets and languages are planned to ensure broader generalizability.
  • Subjective Evaluation: The current evaluation is purely objective. The authors plan to conduct subjective listening tests with both normal-hearing participants using CI simulations and actual CI users to provide a more comprehensive assessment.
  • Hardware Implementation and Latency: The current system is suitable for applications with relaxed timing constraints. Addressing latency for real-time CI deployment is a significant challenge. Future work includes exploring hardware implementation and optimizing the model for reduced computational load.
  • Exploring Other Multisensory Cues: While lip movements are used, future research could investigate other sensory cues to further enhance CI perception.
  • Model Optimization: Investigating lightweight AVSE backbones and exploring techniques like pruning and knowledge distillation are planned to reduce computational overhead.

Summary updated

Abstract

The cochlear implant (CI) is a successful biomedical device that enables individuals with severe-to-profound hearing loss to perceive sound through electrical stimulation, yet listening in noise remains challenging. Recent deep learning advances offer promising potential for CI sound coding by integrating visual cues. In this study, an audio-visual speech enhancement (AVSE) module is integrated with the ElectrodeNet-CS (ECS) model to form the end-to-end CI system, AVSE-ECS. Simulations show that the AVSE-ECS system with joint training achieves high objective speech intelligibility and improves the signal-to-error ratio by 7.4666 dB compared to the advanced combination encoder strategy. These findings underscore the potential of AVSE-based CI sound coding.
BibTeX
@inproceedings{lin2026end,
  author = {Meng-Ping Lin and Enoch Hsin-Ho Huang and Shao-Yi Chien and Yu Tsao},
  title = {{End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments}},
  booktitle = {JASA},
  year = {2026},
  doi = {10.1121/10.0042198},
}
Figure