Abstract
Hand-crafted spatial features, such as inter-channel intensity difference (IID) and inter-channel phase difference (IPD), play a fundamental role in recent deep learning based dual-microphone speech enhancement (DMSE) systems. However, learning the mutual relationship between artificially designed spatial and spectral features is hard in the end-to-end DMSE. In this work, a novel architecture for DMSE using a multi-head cross-attention based convolutional recurrent network (MHCA-CRN) is presented. The proposed MHCA-CRN model includes a channel-wise encoding structure for preserving intra-channel features and a multi-head cross-attention mechanism for fully exploiting cross-channel features. In addition, the proposed approach specifically formulates the decoder with an extra SNR estimator to estimate frame-level SNR under a multi-task learning framework, which is expected to avoid speech distortion led by end-to-end DMSE module. Finally, a spectral gain function is adopted to further suppress the unnatural residual noise. Experiment results demonstrated superior performance of the proposed model against several state-of-the-art models.
Introduction
In the landscape of mobile communication, dual-microphone setups have become the standard for mitigating background noise and improving speech intelligibility. Historically, dual-microphone speech enhancement (DMSE) systems have relied heavily on hand-crafted spatial features, such as Inter-channel Intensity Difference (IID) and Inter-channel Phase Difference (IPD). While these features provide a mathematical basis for distinguishing target speakers from ambient noise, modern deep learning models often struggle to effectively integrate these artificial features with raw spectral data. This "under-utilization" of spatial information, combined with the tendency of end-to-end models to introduce audible speech distortion during aggressive noise suppression, necessitates a more integrated architectural approach.
Proposed Architecture: MHCA-CRN
To address these challenges, we propose the Multi-Head Cross-Attention Convolutional Recurrent Network (MHCA-CRN). The architecture moves away from manual feature engineering, instead allowing the network to learn spatial relationships dynamically through four integrated components:
1. Channel-Wise Encoder
Unlike traditional models that mix microphone signals at the input layer, the MHCA-CRN employs a channel-wise encoding structure. Each microphone signal is processed independently through 2D convolutional layers. This design preserves the unique "intra-channel" characteristics of each signal before they are subjected to cross-channel comparison.
2. Multi-Head Cross-Attention (MHCA)
The core innovation of this model is the MHCA mechanism. Following each downsampling block in the encoder, the feature maps from the primary and reference channels are interchanged and processed via multi-head attention. This allows the model to synchronize time delays and aggregate spatial information across channels dynamically. By calculating attention between channels, the network effectively "learns" the spatial cues that were previously provided by hand-crafted IID and IPD features.
3. Multi-Task Learning with SNR Estimation
To prevent the speech distortion common in end-to-end enhancement, the decoder is formulated under a multi-task learning framework. In addition to predicting the clean spectrum, the model includes an auxiliary branch to estimate the frame-level Signal-to-Noise Ratio (SNR). This branch acts as a regularizer, ensuring the network maintains a high fidelity to the original speech signal while suppressing noise.
4. Spectral Gain Function
As a final post-processing step, the estimated SNR is used to derive a spectral gain function ($G^{SNR}$). This function is applied to the estimated spectrum to suppress "unnatural" residual noise that may have bypassed the main enhancement module, resulting in a cleaner and more natural-sounding output.
Experimental Setup
The model was evaluated using a combination of real-world speech and simulated acoustic environments:
- Data Sources: 30 hours of clean speech from the Librispeech corpus and background noise from the DEMAND dataset.
- Simulation: The IMAGE method was used to simulate Room Impulse Responses (RIRs) for a mobile phone scenario (two microphones with 2cm spacing) in a 5m x 5m x 3m room.
- Conditions: Training and testing were conducted across SNR levels ranging from -5dB to 10dB.
Results and Discussion
The MHCA-CRN was benchmarked against several state-of-the-art models, including the DC-CRN. Performance was measured using the Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) metrics.
Performance Gains
MHCA-CRN consistently outperformed all baseline models across every tested SNR level. For instance, at 0 dB SNR, the proposed model achieved a PESQ of 3.03, significantly higher than the 2.94 achieved by the DC-CRN.
Ablation Study
To verify the impact of the attention mechanism, an ablation study was conducted. Removing the MHCA blocks resulted in a significant performance drop (PESQ fell from 3.03 to 2.64 at 0 dB). This confirms that the cross-attention mechanism is substantially more effective at utilizing spatial information than simple channel concatenation.
Attention Visualization
Visualization of the attention masks revealed that the model learns a hierarchical approach to spatial processing. In the early layers, the model focuses on specific feature channels, while deeper layers focus on synchronizing time delays between the two microphones, effectively performing a learned version of beamforming.
Conclusion
The MHCA-CRN demonstrates that learning cross-channel features through multi-head attention is superior to relying on hand-crafted spatial features for dual-microphone speech enhancement. While the current study focused on a fixed 2cm microphone geometry and simulated RIRs, future work will explore the model's generalization to different device form factors and its computational efficiency for real-time deployment on mobile hardware.
Summary updated