TL;DR
This paper introduces a novel Deformable Convolutional Generative Adversarial Network (GAN) specifically designed to improve speech intelligibility for cochlear implant (CI) users in reverberant environments. By adaptively adjusting its receptive field, the model effectively tackles the unique challenges CI users face with distorted speech signals, leading to significant improvements in both objective and subjective evaluations.
What Problem Does This Paper Address?
Reverberation, the effect of sound reflecting off surfaces in an environment, significantly degrades speech quality and intelligibility. While conventional and deep learning methods have made strides in dereverberation for normal-hearing individuals, these advancements haven't translated as effectively to cochlear implant (CI) users. CI users are particularly susceptible to reverberation due to the nature of their auditory prosthetics, which can struggle to process the smeared and distorted acoustic cues. This leads to a substantial reduction in their ability to understand speech, even in moderately reverberant conditions. The paper aims to bridge this gap by developing a dereverberation technique tailored to the specific needs of CI users.
Methods
The core innovation of this work lies in the integration of deformable convolution networks within a Generative Adversarial Network (GAN) framework.
- Deformable Convolution: Unlike standard convolutions that use a fixed grid of sampling locations, deformable convolutions learn to predict offsets for these locations. This allows the network to dynamically adjust its receptive field to focus on relevant parts of the reverberant speech signal, effectively adapting to varying levels of distortion. This is crucial for capturing important transient speech segments that are often blurred by reverberation.
- GAN Framework: The authors employ a GAN architecture, consisting of a generator and a discriminator. The generator's role is to produce dereverberated speech, while the discriminator is trained to distinguish between real clean speech and the generated dereverberated speech. This adversarial training process pushes the generator to produce more realistic and perceptually superior outputs.
- Model Architecture: The generator includes an encoder, transformer blocks for capturing temporal and frequency dependencies, and a mask decoder. Notably, deformable convolution layers are integrated throughout the encoder and within the mask decoder. The discriminator is also designed to approximate perceptual quality metrics like PESQ.
Datasets
The primary dataset used is the REVERB challenge dataset. This dataset includes:
- Simulated Data (SimData): Clean speech convolved with measured room impulse responses and added noise, across various room sizes, reverberation times, and speaker-microphone distances.
- Real Recordings (RealData): Actual recordings from the MC-WSJ-AV corpus under different conditions.
The paper utilizes the single-channel configuration of this dataset for training, development, and evaluation.
Key Insights and Ideas
- Deformable Convolutions are Key for CI Users: The adaptive nature of deformable convolutions is particularly beneficial for CI users, who require precise extraction of spectro-temporal cues that are heavily distorted by reverberation.
- GANs Enhance Perceptual Quality: The adversarial training of the GAN framework, guided by a metric discriminator, pushes the model beyond simple signal reconstruction to achieve perceptually superior dereverberation, crucial for intelligibility.
- Bridging the Gap for CI Users: This work highlights that methods effective for normal-hearing individuals may not directly translate to CI users, necessitating specialized approaches.
- Envelope Structure Preservation: The proposed method excels at preserving the speech envelope structure, which is vital for CI users to perceive transient speech segments and thus comprehend speech.
Key Results
- Objective Metrics: The proposed method outperforms baseline methods (WPE, BSW, SkipConvNet, SkipConvGAN, CMGAN) on most objective metrics, including PESQ and SRMR, on the REVERB challenge dataset. It shows superior SRMR on real recordings, indicating better robustness.
- Subjective Listening Tests (NH Users): For normal-hearing participants, the proposed method significantly improves intelligibility (WRR of 95.54%) and speech quality (score of 3.4) compared to reverberant speech.
- Subjective Listening Tests (CI Users): This is where the impact is most pronounced. For CI users, the proposed method dramatically boosts intelligibility, increasing Word Recognition Rate (WRR) from 50% (reverberant) to 73.61%, significantly narrowing the gap to clean speech. Speech quality also improved to 4.9.
- Ablation Study: Removing deformable convolutions or the discriminator led to performance degradation, confirming the importance of both components.
Limitations and Future Work
- Real-time Performance: The authors acknowledge that future work will explore causal or compressed architectures to improve real-time efficiency and reduce computational complexity, making the model more practical for CI applications.
- Generalization: While tested on the REVERB dataset, further evaluation on diverse real-world scenarios and different CI devices could be beneficial.
Summary updated