Adaptive beamforming and soft missing data decoding for robust speech recognition in reverberant environments

Marco Kühne, Roberto Togneri, Sven NordholmInterspeech 2008

TL;DR

This paper tackles the challenge of robust speech recognition in noisy, reverberant environments with multiple speakers. They propose a novel system that combines adaptive beamforming with a soft missing data decoding approach, significantly improving recognition accuracy compared to existing methods.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems struggle in real-world scenarios like busy cafes or meeting rooms where multiple people are talking and sound bounces off surfaces (reverberation). These conditions make it difficult for ASR to isolate the target speaker's voice and accurately interpret the speech. Traditional ASR systems often fail because they are trained on clean, anechoic (echo-free) speech. While some methods use microphone arrays to separate speakers or adapt ASR models to reverberant conditions, these approaches have limitations, especially when dealing with both multiple speakers and significant reverberation. This paper aims to bridge this gap by developing a system that can effectively handle both challenges simultaneously.

Methods

The proposed system has two main components:

  • Source Separation and Localization:
    • It uses a microphone array to capture audio.
  • Spatial cues (like the direction of arrival or DOA) are extracted from the microphone signals.
  • A novel fuzzy c-means clustering algorithm (sFCM) is employed to cluster these DOAs. This algorithm is adapted to incorporate neighborhood information of time-frequency (TF) points, making the resulting masks more robust to noise and reverberation.
  • These clustered DOAs and TF masks are then used to blindly design multiple adaptive beamformers. These beamformers aim to enhance the desired speaker's signal while suppressing others.
  • Soft Missing Data Decoding:
    • The enhanced speech signal is fed into a missing data speech recognition (MD-ASR) system.
  • A key innovation here is a new "evidence model" that quantifies the reliability of each feature component. This model uses a Gauss-Uniform mixture, allowing for more flexible representation of feature uncertainty.
  • Crucially, this evidence model incorporates both localization information (from the beamforming stage) and feature uncertainty, providing a more robust representation of the speech signal.
  • The system uses a soft Viterbi decoding algorithm, which is well-suited for handling missing or unreliable feature information.

The entire system operates in an unsupervised manner, meaning it doesn't require pre-trained models for source localization or echoic speech.

Datasets

  • Training: TIDigits database (clean anechoic data for training cepstral and log-spectral models).
  • Evaluation: TIDigits database (for connected digit experiments) mixed with a male interfering speaker from the TIMIT database.

The paper notes that the acoustic features were extracted from a mel-filterbank with 32 channels due to a change in sampling frequency to 8 kHz.

Key Insights and Ideas

  • Neighborhood Information for Robust Masks: Incorporating the correlation between adjacent time-frequency points into the clustering process significantly improves the robustness of the localization masks, especially in reverberant conditions. This is a key improvement over standard FCM.
  • Blind Adaptive Beamforming: Using the estimated source DOAs and TF masks to blindly design adaptive beamformers allows for effective enhancement of target speakers without prior knowledge of their locations or characteristics.
  • Gauss-Uniform Evidence Model: The proposed Gauss-Uniform evidence model, with its ability to adapt feature variances, offers a more flexible and potentially more accurate representation of speech signal reliability in mismatched (reverberant) conditions compared to simpler models.
  • Integration of Localization and Feature Uncertainty: The novel evidence model's ability to combine information from both the spatial localization stage and the feature uncertainty estimation is crucial for achieving robust recognition.

Key Results

  • The proposed system significantly outperformed a single-channel baseline and a traditional adaptive beamformer alone, especially as reverberation time increased.
  • The combination of adaptive beamforming and the soft missing data decoder achieved a recognition accuracy of 80.00% at a reverberation time of 300 ms and 64.83% at 450 ms, using the Gauss-Uniform evidence model.
  • The Gauss-Uniform evidence model consistently provided moderate performance improvements over the Dirac-Uniform model (a related model used in many MD-ASR decoders).
  • When compared to a related binaural separation model (from reference [4]), the proposed system showed better performance, particularly at higher reverberation times.

Limitations and Future Work

  • Limitations:
    • The performance of the adaptive beamformer degrades significantly beyond 200 ms reverberation time, as the sparseness assumption and localization cues become unreliable.
  • The accuracy of the uncertainty estimation and the effectiveness of its utilization depend on the strength of the estimated mixture weight.
  • The paper notes that there is still room for improvement, as indicated by the performance gap between the proposed system and the "oracle" evidence models (which use perfect knowledge of the target signal).
  • Future Work:
    • Making the beamformer design more robust against mismatches in steering vector and jammer correlation matrix.
  • Complementing the localization cues with non-spatial features (like harmonicity) that are known to be more resistant to reverberation.
  • Developing better methods for estimating mixture weights and feature variances to close the gap with oracle models.

Summary updated

BibTeX
@inproceedings{kuhne2008adaptive,
  author = {Marco Kühne and Roberto Togneri and Sven Nordholm},
  title = {{Adaptive beamforming and soft missing data decoding for robust speech recognition in reverberant environments}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure