Speech recognition with a hearing-aid processing scheme combining beamforming with mask-informed speech enhancement

Tim Green, Gaston Hilkhuysen, Mark Huckvale, Stuart Rosen, Mike Brookes, Alastair H. Moore, Patrick A. Naylor, Leo Lightburn, Wei XueTrends 2022 · DOI

TL;DR

This study investigates combining beamforming with mask-informed speech enhancement for hearing aids in noisy environments. They found that binaural beamforming significantly improved speech recognition for hearing-impaired listeners compared to bilateral beamforming, but adding mask-informed enhancement actually decreased performance.

What Problem Does This Paper Address?

Understanding speech in noisy environments is a major challenge for individuals with hearing impairments, significantly impacting their satisfaction with and use of hearing aids. Existing solutions often involve spatial filtering (beamforming) to focus on the target sound source and signal processing techniques to separate speech from noise based on its characteristics. While these methods show promise in controlled settings, their effectiveness in real-world, dynamic listening situations (like a noisy classroom) is less clear. This paper aims to evaluate a specific combination of advanced signal processing techniques – binaural and bilateral beamforming coupled with a deep neural network-based speech enhancement – to see if it can overcome these challenges and improve speech recognition for hearing-impaired listeners.

Methods

The researchers compared two types of beamforming:

  • Binaural Beamforming: Treats the microphones from both hearing aids as a single, larger array. This creates a narrower beam, maximizing the signal-to-noise ratio (SNR) but potentially losing spatial information in the residual noise.
  • Bilateral Beamforming: Uses independent beamformers for each hearing aid. This preserves spatial diversity in the residual noise but results in a broader beam and less SNR improvement.

Both beamformer types were optimized for the target speech direction. They then evaluated these beamformers with and without an additional "mask-informed speech enhancement" module. This enhancement module used a deep neural network (DNN) trained to identify time-frequency regions dominated by target speech. For normally-hearing listeners, they also tested two other enhancement methods: Wiener filtering and modulation-domain Kalman filtering.

The study used a simulated classroom environment to mimic realistic adverse listening conditions. Speech reception thresholds (SRTs) were measured for hearing-impaired (HI) and normally-hearing (NH) listeners under various processing conditions.

Datasets

The study utilized a database of binaural impulse responses from Kayser et al. (2009) to simulate anechoic environments. For the classroom environment, they recorded ambient noise and binaural room impulse responses (BRIRs) in a real primary school classroom using a KEMAR manikin equipped with dummy hearing aids. The target speech material consisted of IEEE sentences.

Key Insights and Ideas

  • Binaural Beamforming Wins: For hearing-impaired listeners, binaural beamforming consistently outperformed bilateral beamforming, even when the latter preserved more spatial information. This suggests that maximizing SNR through a narrower beam was more critical than preserving spatial cues in this specific noisy classroom scenario.
  • Enhancement Hinders: Surprisingly, adding the mask-informed speech enhancement to either beamforming type reduced speech recognition performance for both HI and NH listeners. This indicates that the enhancement, while designed to isolate speech, might have negatively impacted the benefits provided by the beamformers.
  • NH vs. HI Similarities: The general patterns of results were similar for both normally-hearing and hearing-impaired listeners, suggesting that the observed effects of beamforming and enhancement are not solely attributable to the hearing impairment itself.
  • Real-World Simulation Matters: The use of a simulated classroom environment, with its complex and dynamic noise, is highlighted as crucial for evaluating hearing aid technologies, as results can differ significantly from simpler laboratory conditions.

Key Results

  • Significant Improvement with Beamforming: Both binaural and bilateral beamforming provided substantial improvements in SRT compared to no processing for HI listeners in the classroom. Binaural beamforming offered a greater benefit, with SRTs improving by around 4-6 dB compared to unprocessed conditions.
  • Enhancement Detriment: Adding mask-informed enhancement to beamforming resulted in poorer performance. For binaural beamforming, this led to an increase in SRT of 0.3-1.2 dB for HI listeners and 1.5-2.2 dB for NH listeners.
  • Anechoic vs. Classroom: In anechoic conditions, the mask-informed enhancement was beneficial for NH listeners when noise was spatially separated, but not for HI listeners. The study's primary focus on the classroom environment showed a consistent detriment from this enhancement.
  • Alternative Enhancements: For NH listeners, Wiener and Kalman filtering, when combined with binaural beamforming, performed better than the mask-informed DNN approach, though they did not improve performance beyond beamforming alone.

Limitations and Future Work

  • Optimized Beamformers: The study used beamformers optimized for the target speech direction. The authors suggest future work should investigate performance when beamformers are not perfectly adapted, as this might reveal a greater benefit for mask-informed enhancement.
  • Training Data Mismatch: A potential limitation for the mask-informed enhancement is that the training data for the DNN might not have been sufficiently similar to the actual testing noise. Future research could explore training with a wider variety of noise types.
  • Spatial Cue Preservation: The method used to preserve spatial cues in the enhancement process (applying the same gain to both channels) might degrade these cues. Further investigation into this aspect is recommended.
  • Real-World Dynamics: The study acknowledges that real-world listening involves head movements, dynamic talker changes, and visual cues, which were not fully simulated. Future work could explore these factors.
  • Alternative Enhancement Integration: The authors suggest exploring other methods for integrating spatial filtering and speech enhancement.

Summary updated

Abstract

A signal processing approach combining beamforming with mask-informed speech enhancement was assessed by measuring sentence recognition in listeners with mild-to-moderate hearing impairment in adverse listening conditions that simulated the output of behind-the-ear hearing aids in a noisy classroom. Two types of beamforming were compared: binaural, with the two microphones of each aid treated as a single array, and bilateral, where independent left and right beamformers were derived. Binaural beamforming produces a narrower beam, maximising improvement in signal-to-noise ratio (SNR), but eliminates the spatial diversity that is preserved in bilateral beamforming. Each beamformer type was optimised for the true target position and implemented with and without additional speech enhancement in which spectral features extracted from the beamformer output were passed to a deep neural network trained to identify time-frequency regions dominated by target speech. Additional conditions comprising binaural beamforming combined with speech enhancement implemented using Wiener filtering or modulation-domain Kalman filtering were tested in normally-hearing (NH) listeners. Both beamformer types gave substantial improvements relative to no processing, with significantly greater benefit for binaural beamforming. Performance with additional mask-informed enhancement was poorer than with beamforming alone, for both beamformer types and both listener groups. In NH listeners the addition of mask-informed enhancement produced significantly poorer performance than both other forms of enhancement, neither of which differed from the beamformer alone. In summary, the additional improvement in SNR provided by binaural beamforming appeared to outweigh loss of spatial information, while speech understanding was not further improved by the mask-informed enhancement method implemented here.
BibTeX
@inproceedings{green2022speech,
  author = {Tim Green and Gaston Hilkhuysen and Mark Huckvale and Stuart Rosen and Mike Brookes and Alastair H. Moore and Patrick A. Naylor and Leo Lightburn and Wei Xue},
  title = {{Speech recognition with a hearing-aid processing scheme combining beamforming with mask-informed speech enhancement}},
  booktitle = {Trends},
  year = {2022},
  doi = {10.1177/23312165211068629},
}
Figure