Introduction
State-of-the-art multi-channel speech recognition relies on neural networks to separate speech from noise. However, these models typically require paired clean and noisy audio for training—a requirement that is difficult to meet with real-world data. This paper introduces an unsupervised method that leverages classical multi-channel linear prediction (MCLP) to generate "pseudo-targets," enabling neural mask estimators to train directly on real-world noisy data. The resulting system matches the performance of supervised models trained on perfect simulated data and significantly outperforms models trained on out-of-domain data.
Problem Statement
In multi-channel Automatic Speech Recognition (ASR), beamforming spatially filters audio to enhance target speech while suppressing background noise. Modern beamformers, such as Generalized Eigen-Value (GEV) beamformers, rely on Deep Neural Networks (DNNs) to estimate "masks" that identify speech versus noise in time-frequency bins.
Training these DNNs usually requires parallel datasets: identical recordings in both perfectly clean and noisy/reverberant conditions. Since recording such parallel data in the real world is impossible, researchers often rely on simulated data. This creates a domain mismatch when systems are deployed in complex real-world environments. This research addresses the challenge of training neural mask estimators in an unsupervised manner, directly on real noisy recordings without clean reference tracks.
Methodology
The proposed approach uses a bootstrapping technique that combines classical signal processing with deep learning to bypass the need for ground-truth clean audio.
- Target Generation via MCLP: The system employs a joint spatial filtering and Multi-Channel Linear Prediction (MCLP) approach. MCLP mathematically models late reverberation and estimates the clean source signal using Bayesian inference.
- Pseudo-Target Creation: The estimated clean signal from the MCLP step is used to compute an Ideal Ratio Mask (IRM). This mask serves as an unsupervised "pseudo-target" for the neural network.
- Neural Mask Estimation: A Bi-directional Long Short-Term Memory (BLSTM) network is trained using these pseudo-targets to predict speech presence probability for each time-frequency bin.
- GEV Beamforming: The DNN-predicted masks calculate the Power Spectral Density (PSD) matrices for speech and noise. These are fed into a GEV beamformer to produce enhanced audio for the ASR acoustic model.
Experimental Setup
The approach was evaluated on two multi-channel ASR benchmarks:
- CHiME-3: Multi-microphone tablet recordings in everyday environments (cafe, street, transport, pedestrian area), including both simulated and real-world noisy data.
- REVERB Challenge: 8-channel recordings with real and simulated reverberant conditions.
Key Results
The unsupervised approach effectively closes the gap between simulation and reality. On the CHiME-3 dataset, the unsupervised GEV approach achieved an average Word Error Rate (WER) of 8.1% on the evaluation set, outperforming the supervised oracle GEV model (8.3% WER) and significantly surpassing supervised models trained on out-of-domain data (10.9% WER).
On the REVERB Challenge dataset, the unsupervised method achieved an average evaluation WER of 9.4%, performing on par with the supervised oracle model (9.2% WER). Compared to the standard BeamformIt baseline, the proposed method yielded relative WER improvements of 25% to 37% across the evaluation datasets.
Discussion and Conclusion
This work demonstrates that classical signal processing can effectively supervise deep learning. By using MCLP to generate useful targets, neural networks can learn high-quality representations without human-annotated or perfectly simulated ground truth. Crucially, in-domain unsupervised training proved superior to out-of-domain supervised training, highlighting the value of training on the specific acoustic environments the system will encounter in deployment.
While the system shows robust performance, some degradation was noted in specific simulated noise conditions, such as "Street" noise in CHiME-3, where MCLP pseudo-targets may struggle with highly non-stationary noise. Future research may focus on refining pseudo-target generation for dynamic environments or exploring joint training of the mask estimator and the ASR model.
Summary updated