TL;DR
This study investigates combining beamforming with mask-informed speech enhancement for hearing aids in noisy environments. They found that binaural beamforming significantly improved speech recognition for hearing-impaired listeners compared to bilateral beamforming, but adding mask-informed enhancement actually decreased performance.
What Problem Does This Paper Address?
Understanding speech in noisy environments is a major challenge for individuals with hearing impairments, significantly impacting their satisfaction with and use of hearing aids. Existing solutions often involve spatial filtering (beamforming) to focus on the target sound source and signal processing techniques to separate speech from noise based on its characteristics. While these methods show promise in controlled settings, their effectiveness in real-world, dynamic listening situations (like a noisy classroom) is less clear. This paper aims to evaluate a specific combination of advanced signal processing techniques – binaural and bilateral beamforming coupled with a deep neural network-based speech enhancement – to see if it can overcome these challenges and improve speech recognition for hearing-impaired listeners.
Methods
The researchers compared two types of beamforming:
- Binaural Beamforming: Treats the microphones from both hearing aids as a single, larger array. This creates a narrower beam, maximizing the signal-to-noise ratio (SNR) but potentially losing spatial information in the residual noise.
- Bilateral Beamforming: Uses independent beamformers for each hearing aid. This preserves spatial diversity in the residual noise but results in a broader beam and less SNR improvement.
Both beamformer types were optimized for the target speech direction. They then evaluated these beamformers with and without an additional "mask-informed speech enhancement" module. This enhancement module used a deep neural network (DNN) trained to identify time-frequency regions dominated by target speech. For normally-hearing listeners, they also tested two other enhancement methods: Wiener filtering and modulation-domain Kalman filtering.
The study used a simulated classroom environment to mimic realistic adverse listening conditions. Speech reception thresholds (SRTs) were measured for hearing-impaired (HI) and normally-hearing (NH) listeners under various processing conditions.
Datasets
The study utilized a database of binaural impulse responses from Kayser et al. (2009) to simulate anechoic environments. For the classroom environment, they recorded ambient noise and binaural room impulse responses (BRIRs) in a real primary school classroom using a KEMAR manikin equipped with dummy hearing aids. The target speech material consisted of IEEE sentences.
Key Insights and Ideas
- Binaural Beamforming Wins: For hearing-impaired listeners, binaural beamforming consistently outperformed bilateral beamforming, even when the latter preserved more spatial information. This suggests that maximizing SNR through a narrower beam was more critical than preserving spatial cues in this specific noisy classroom scenario.
- Enhancement Hinders: Surprisingly, adding the mask-informed speech enhancement to either beamforming type reduced speech recognition performance for both HI and NH listeners. This indicates that the enhancement, while designed to isolate speech, might have negatively impacted the benefits provided by the beamformers.
- NH vs. HI Similarities: The general patterns of results were similar for both normally-hearing and hearing-impaired listeners, suggesting that the observed effects of beamforming and enhancement are not solely attributable to the hearing impairment itself.
- Real-World Simulation Matters: The use of a simulated classroom environment, with its complex and dynamic noise, is highlighted as crucial for evaluating hearing aid technologies, as results can differ significantly from simpler laboratory conditions.
Key Results
- Significant Improvement with Beamforming: Both binaural and bilateral beamforming provided substantial improvements in SRT compared to no processing for HI listeners in the classroom. Binaural beamforming offered a greater benefit, with SRTs improving by around 4-6 dB compared to unprocessed conditions.
- Enhancement Detriment: Adding mask-informed enhancement to beamforming resulted in poorer performance. For binaural beamforming, this led to an increase in SRT of 0.3-1.2 dB for HI listeners and 1.5-2.2 dB for NH listeners.
- Anechoic vs. Classroom: In anechoic conditions, the mask-informed enhancement was beneficial for NH listeners when noise was spatially separated, but not for HI listeners. The study's primary focus on the classroom environment showed a consistent detriment from this enhancement.
- Alternative Enhancements: For NH listeners, Wiener and Kalman filtering, when combined with binaural beamforming, performed better than the mask-informed DNN approach, though they did not improve performance beyond beamforming alone.
Limitations and Future Work
- Optimized Beamformers: The study used beamformers optimized for the target speech direction. The authors suggest future work should investigate performance when beamformers are not perfectly adapted, as this might reveal a greater benefit for mask-informed enhancement.
- Training Data Mismatch: A potential limitation for the mask-informed enhancement is that the training data for the DNN might not have been sufficiently similar to the actual testing noise. Future research could explore training with a wider variety of noise types.
- Spatial Cue Preservation: The method used to preserve spatial cues in the enhancement process (applying the same gain to both channels) might degrade these cues. Further investigation into this aspect is recommended.
- Real-World Dynamics: The study acknowledges that real-world listening involves head movements, dynamic talker changes, and visual cues, which were not fully simulated. Future work could explore these factors.
- Alternative Enhancement Integration: The authors suggest exploring other methods for integrating spatial filtering and speech enhancement.
Summary updated