Abstract
This study proposes a novel all-neural approach for multi-channel speech enhancement, where robust speaker localization, acoustic beamforming, post-filtering and spatial filtering are all done using deep learning based time-frequency (T-F) masking. Our system first performs monaural speech enhancement on each microphone signal to obtain the estimated ideal ratio masks for beamforming and robust time delay of arrival (TDOA) estimation. Then with the estimated TDOA, directional features indicating whether each T-F unit is dominated by the signal coming from the estimated target direction are computed. Next, the directional features are combined with the spectral features extracted from the beamformed signal to achieve further enhancement. Experiments on a two-microphone setup in reverberant environments with strong diffuse babble noise demonstrate the effectiveness of the proposed approach for multi-channel speech enhancement.
Introduction
Enhancing speech using multiple microphones is a classic challenge in signal processing. Traditionally, this relies on beamforming: designing a linear filter that boosts sound from a target direction while suppressing noise from others. However, traditional beamforming has two major "Achilles' heels":
- Localization Sensitivity: It requires an accurate Direction of Arrival (DOA) or Time Delay of Arrival (TDOA). Standard algorithms like GCC-PHAT often fail in reverberant rooms or when noise is non-stationary.
- Limited Suppression: Linear beamforming alone often cannot suppress enough noise, especially with only two microphones or in "diffuse" noise environments, such as a crowded cafeteria.
This research bridges the gap between powerful single-channel deep learning models and multi-channel spatial processing to create a system robust to both echoes and heavy background babble.
System Architecture
The proposed system is a multi-stage pipeline where every step is driven by Deep Neural Networks, specifically Bidirectional LSTMs (BLSTMs).
- Neural Masking for Statistics: A BLSTM performs single-channel enhancement on each microphone to estimate Ideal Ratio Masks (IRM). These masks identify which T-F units are speech-heavy versus noise-heavy.
- Neural-Driven MVDR Beamforming: Instead of guessing noise statistics, the system uses the estimated masks to compute speech and noise covariance matrices. These matrices are then used to build a Minimum Variance Distortionless Response (MVDR) beamformer.
- Robust TDOA Estimation: The system extracts "steering vectors" from the neural-driven beamformer and searches for a time delay that best matches the phase differences across all frequencies, weighted by the speech mask to ignore noisy regions.
- Spatial + Spectral Post-Filtering: Finally, the system computes Directional Features (DF)—indicating if a specific sound unit comes from the target direction—and feeds them along with the beamformed audio into a final BLSTM for the final cleanup.
Experimental Setup
The researchers evaluated the system using a simulated two-microphone setup in a virtual 8x8x3m room. The dataset included 720 IEEE female utterances as target speech, with diffuse babble noise created by mixing 37 different TIMIT speakers. Room Impulse Responses (RIRs) were generated with reverberation times ($T_{60}$) ranging from 0.0s to 1.0s. The model was trained on 36 specific directions and tested on 37 unseen directions to ensure generalization.
Key Results and Discussion
The "all-neural" approach showed dramatic improvements over traditional baselines across several metrics:
- Localization Accuracy: The proposed TDOA estimation achieved 92.0% gross accuracy, compared to a meager 25.8% for the standard GCC-PHAT baseline in the same conditions. This resilience to reverberation is a core strength of the neural-weighted spectrum approach.
- Intelligibility (STOI): Unprocessed noisy audio had a score of 48.5%. The full multi-channel neural system boosted this to 74.4%.
- Quality (PESQ): Speech quality improved from a baseline of 0.98 to 2.05, outperforming both single-channel neural models and oracle (perfect) linear beamformers.
A key insight is that T-F masking serves as a powerful statistical estimator. By masking out noise before calculating covariance, the beamformer becomes much more precise. Furthermore, by combining spatial features (DF) with spectral features, the final BLSTM can distinguish between a target speaker and a noise source even if they are located at the same angle.
Conclusion
This study demonstrates that replacing traditional signal processing components with deep learning-based T-F masking significantly enhances multi-channel speech processing robustness. Future work may involve extending the framework to larger microphone arrays, handling multiple simultaneous target speakers, and optimizing the computational load for mobile devices.
Summary updated