Abstract
Far-field Automatic Speech Recognition (ASR) faces significant performance degradation due to background noise and reverberation. While traditional spatial beamforming techniques like MVDR are commonly used, they often fail when signal-to-noise ratios (SNR) vary drastically across different microphones in an array. This paper proposes a feature-domain approach that intelligently selects or weights microphone channels. By introducing a novel Jacobian constraint to an Expectation-Maximization (EM) weighting framework, the authors demonstrate that maintaining feature variance is critical for robust recognition. Experimental results on the CHiME-3 dataset show that the proposed weighting method significantly outperforms both standard beamforming and hard channel selection, particularly in real-world environments with mismatched channel qualities.
Introduction
In far-field ASR, the distance between the speaker and the microphone array introduces acoustic challenges. Standard solutions typically rely on beamforming to spatially filter audio. However, beamforming assumes relatively uniform signal quality across the array. In practical applications—such as a mobile device where one microphone might be obstructed or facing a noise source—this assumption breaks down. When a high-noise channel is mixed into a beamformer, it can degrade the final output to a level worse than a single-channel baseline. This research investigates methods to combine 40-dimensional log Mel filterbank features directly, bypassing the need for spatial phase information and focusing on signal quality optimization.
Methodology
The authors evaluate two primary strategies for multi-channel feature fusion: Channel Selection and Channel Weighting.
Channel Selection
Selection aims to identify the single highest-quality microphone channel. Two methods are explored:
- Maximum Likelihood (ML): A Gaussian Mixture Model (GMM) is trained on clean speech. During testing, the channel that yields the highest likelihood against this GMM is selected.
- Autoencoder (AE): An autoencoder is trained to reconstruct clean speech from clean inputs. In the testing phase, noisy channels are fed through the AE; the channel resulting in the lowest reconstruction error (the difference between input and output) is chosen as the best representative of the speech signal.
Channel Weighting and the Jacobian Constraint
Rather than discarding data, channel weighting uses a linear weighted sum of all features. Weights are estimated via an EM algorithm to maximize likelihood against a clean-speech GMM. However, simple likelihood maximization often leads to "regression-to-the-mean," where the combined features lose variance and become "flat," hurting the acoustic model's performance.
To counteract this, the authors introduce a Jacobian Constraint. By adding the log determinant of the covariance matrix of the weighted features to the objective function, the model is mathematically forced to maintain a healthy feature variance. This ensures the fused features remain recognizable to the neural network-based acoustic model.
Experimental Setup
The proposed methods were tested using the 3rd CHiME Speech Separation and Recognition Challenge (CHiME-3) dataset. This dataset is particularly challenging because it includes a 6-channel array recorded in diverse real-world settings (cafes, buses, streets). Notably, Channel 2 in the real-world recordings often captures significantly more noise than others, providing a rigorous test for algorithms sensitive to channel mismatch.
Results and Discussion
The experimental results highlight the limitations of traditional spatial processing in mismatched conditions. On the real-world development set, standard MVDR beamforming across all six channels resulted in a Word Error Rate (WER) of 12.55%. In contrast, the Jacobian-constrained weighting method achieved a superior WER of 9.97%.
As shown in the spectrogram comparisons above, the Jacobian-constrained features preserve the dynamic range and phonetic structure of the speech signal much more effectively than simple channel selection. While selection methods (ML and AE) performed reasonably well (approx. 11.5% WER), they could not match the performance of the weighting approach, which successfully extracted useful information even from noisier channels while heavily penalizing the worst ones.
Conclusion
This paper demonstrates that feature-domain channel weighting with variance preservation is a powerful alternative to spatial beamforming for robust ASR. The Jacobian constraint is essential for preventing feature distortion during the fusion process. Future work will explore frequency-specific weighting, allowing the system to dynamically prioritize different microphones for different frequency bands, further enhancing robustness against colored noise sources.
Summary updated