TL;DR
This paper introduces a novel deep learning-based mel-subband beamformer for in-car speech separation that significantly reduces computational cost and inference time compared to existing methods. By processing audio in non-uniform subbands based on a mel-scale, it achieves performance comparable to narrow-band systems while being much more efficient, making it suitable for real-world automotive applications.
What Problem Does This Paper Address?
In-car speech separation is crucial for enhancing human-machine interaction in vehicles. Existing deep learning (DL) approaches, while effective, often struggle with high computational costs and slow inference times, especially when processing narrow-band (NB) frequencies independently. Full-band (FB) systems, which process all frequencies at once, tend to suffer from overfitting and reduced performance. Subband (SB) approaches offer a compromise but typically divide the spectrum into uniform bands, which can be suboptimal for speech, particularly in lower frequencies where crucial formant structures reside. This paper aims to bridge this gap by developing an efficient and effective in-car speech separation system that balances performance with computational constraints.
Methods
The proposed system is a deep neural mel-subband spatio-temporal RNN beamformer (SRNN-BF) that operates end-to-end. It consists of three main modules:
- Multi-channel Feature Extraction: This module takes multi-channel audio mixtures and a single-channel in-car loudspeaker reference signal. It uses 1D convolutional layers with Short-Time Fourier Transform (STFT) operations to extract features like log power spectrum (LPS), inter-channel phase difference (IPD), and directional angle features (DF).
- Mask Estimation Module: This module predicts speech and noise masks to identify time-frequency bins dominated by the target source or noise. It utilizes complex-valued ratio filters (cRFs) to estimate spatial covariance matrices (SCMs) for both speech and noise. Crucially, it also incorporates echo reference signals to jointly model cross-correlations and suppress loudspeaker echoes, a significant improvement over previous methods.
- Subband Beamformer: This is the core innovation. Instead of uniform subbands, it employs a mel-scale based subband selection strategy. This means lower frequencies, which are richer in speech content, are processed with higher resolution, while higher frequencies are processed with coarser resolution. Learnable analysis filters are used to project these non-uniform subbands into a lower-dimensional space. The beamformer weights are then predicted using a unified RNN-DNN layer shared across subbands, with multi-head self-attention (MHSA) for dynamic adaptation. Finally, a mel-scale subband synthesis module converts these SB weights back to full-band weights.
Datasets
- Simulated Multi-speaker In-car Dataset: Generated using AISHELL-2 and AEC-Challenge corpora. This dataset includes 10,000 multi-channel Room Impulse Responses (RIRs) simulating various vehicle dimensions, and incorporates clean and nonlinear loudspeaker distortion functions. It also includes diffused noise with a wide range of SNRs and signal-to-echo ratios (SER).
- Training, Development, and Test Sets: The simulated dataset is split into 180,000 utterances for training, 7,500 for development, and 2,000 for testing.
- Real-world Recordings: Used for evaluation, though specific details on their origin beyond "actual in-car recordings" are not provided.
Key Insights and Ideas
- Mel-Scale Subband Division is Key: The most significant insight is that dividing the audio spectrum into non-uniform subbands based on the mel-scale, rather than uniform bands, dramatically improves performance. This aligns processing resolution with the perceptual importance of different frequency ranges for speech.
- Joint Echo Cancellation and Beamforming: Integrating echo reference signals directly into the SCM estimation process effectively suppresses loudspeaker echoes, a common problem in in-car systems.
- Efficiency without Sacrificing Performance: The mel-subband approach allows for a substantial reduction in computational cost and inference time, making it practical for embedded automotive systems, without a significant drop in speech separation quality.
- NB Performance with SB Efficiency: The proposed mel-subband system achieves performance metrics (like WER) that are very close to traditional narrow-band systems, which are known for their high accuracy, but with a fraction of the computational load.
Key Results
- Improved Performance: The proposed mel-subband beamformer consistently outperforms traditional subband approaches across objective metrics (PESQ, Si-SNR, SDR) and Word Error Rate (WER).
- WER Reduction: It achieves a relative improvement of 30.74% in WER over traditional 64-subband approaches on simulated data and 17.96% on real-world data, with only a marginal increase in computation.
- Comparison to NB: The mel-subband system achieves WER comparable to NB-based GRNNBF (5.53 vs 5.21 simulated, 9.35 vs 8.69 real), while being significantly more computationally efficient (5x faster).
- Outperforms FB: It significantly outperforms all FB-based systems, with an average 3x increase in computing cost for FB systems.
- Computational Efficiency: The system requires significantly lower computing cost and memory usage compared to NB systems, making it suitable for CPU deployment.
Limitations and Future Work
- Limited to Reverberant Speech: The paper notes that the proposed system is limited to estimating reverberant clean speech, not anechoic speech.
- Focus on In-Car: While the system is designed for in-car environments, its applicability to other noisy acoustic scenarios might require further investigation.
- Future Work: The authors suggest that further exploration of different subband splitting strategies and learnable filter designs could lead to even greater improvements. The paper also implicitly points towards the potential for real-time deployment and integration into automotive HMI systems.
Summary updated