Improving Dual-Microphone Speech Enhancement by Learning Cross-Channel Features with Multi-Head Attention

Xinmeng Xu, Rongzhi Gu, Yuexian Zou

TL;DR

This paper introduces MHCA-CRN, a novel deep-learning architecture for dual-microphone speech enhancement that replaces hand-crafted spatial features with a Multi-Head Cross-Attention (MHCA) mechanism. By combining channel-independent encoding with a multi-task learning framework that includes an auxiliary SNR estimator, the model achieves state-of-the-art performance in suppressing noise while minimizing speech distortion.

What Problem Does This Paper Address?

In mobile devices, dual-microphone setups are standard for reducing background noise. Traditionally, researchers have relied on hand-crafted spatial features like Inter-channel Intensity Difference (IID) and Inter-channel Phase Difference (IPD) to help models distinguish between the target speaker and noise.

However, the authors identify a significant bottleneck: standard Deep Neural Networks (DNNs) struggle to learn the complex, mutual relationship between these artificial spatial features and raw spectral data. This often leads to "under-utilization" of spatial information. Furthermore, end-to-end enhancement models frequently introduce audible speech distortion while trying to aggressively suppress noise.

Methods

The authors propose the Multi-Head Cross-Attention Convolutional Recurrent Network (MHCA-CRN). Its architecture consists of four main components:

  1. Channel-Wise Encoder: Instead of mixing the two microphone signals immediately, the model processes each channel independently using 2D convolutional layers. This preserves "intra-channel" features (the unique characteristics of each signal) before they are compared.
  2. Multi-Head Cross-Attention (MHCA): This is the "secret sauce." After each downsampling block in the encoder, the feature maps from the two channels are interchanged via MHCA. This mechanism synchronizes time delays and aggregates spatial information across channels dynamically, effectively "learning" the spatial cues that were previously hand-crafted.
  3. SNR Estimator (Multi-task Learning): The decoder doesn't just predict the clean spectrum; it also includes an auxiliary branch to estimate the frame-level Signal-to-Noise Ratio (SNR). This multi-task approach acts as a regularizer, helping the model maintain speech quality.
  4. Spectral Gain Function: Finally, the estimated SNR is used to calculate a gain function ($G^{SNR}$) that is applied to the estimated spectrum. This post-processing step helps suppress "unnatural" residual noise that the main network might miss.

Datasets

The researchers evaluated their model using a combination of real-world speech and simulated environments:

  • Speech: 30 hours of clean speech from the Librispeech corpus.
  • Noise: Background noises from the DEMAND dataset.
  • Simulation: They used the IMAGE method to simulate Room Impulse Responses (RIRs) for a mobile phone scenario (two microphones with 2cm spacing) in a 5m x 5m x 3m room.
  • Conditions: Training and testing covered SNR levels ranging from -5dB to 10dB.

Key Insights and Ideas

  • Attention as a Spatial Synchronizer: The MHCA module essentially learns to align the two channels. By calculating attention between the primary and reference microphones, the model can "focus" on specific feature map channels and time frames where the target speech is most prominent.
  • Preservation vs. Aggregation: By using independent encoders first and then cross-attention, the model avoids "muddying" the signals too early, allowing it to extract cleaner individual features before performing spatial fusion.
  • The SNR "Safety Net": Directly predicting a clean spectrum can be "greedy" and cause distortion. Using an SNR estimator to derive a gain function provides a more mathematically grounded way to suppress noise without eating into the speech signal.

Key Results

The MHCA-CRN was compared against several baselines, including the previous state-of-the-art DC-CRN. Performance was measured using PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility):

  • Superiority: MHCA-CRN consistently outperformed all baselines across all SNR levels. For example, at 0 dB SNR, it achieved a PESQ of 3.03 compared to DC-CRN's 2.94.
  • Ablation Success: Removing the MHCA blocks caused a significant drop in performance (PESQ dropped from 3.03 to 2.64 at 0 dB), proving that the cross-attention mechanism is far more effective than simple concatenation of channels.
  • Visual Proof: Visualization of the attention masks showed that in early layers, the model focuses on specific feature channels, while in deeper layers, it learns to synchronize time delays between the two microphones.

Limitations and Future Work

While the results are impressive, the paper leaves a few stones unturned:

  • Fixed Geometry: The experiments assumed a fixed 2cm microphone spacing. It is unclear how the model would generalize to different device form factors (e.g., tablets or laptops with wider spacing) without retraining.
  • Computational Complexity: While the authors mention conventional methods are "lightweight," they do not provide a detailed breakdown of the computational overhead (FLOPs/latency) added by the multi-head attention mechanism, which is critical for real-time mobile deployment.
  • Real-world RIRs: The study relies on simulated room impulses; testing on recordings from actual physical rooms with complex reflections would be a logical next step.

Summary updated

Abstract

Hand-crafted spatial features, such as inter-channel intensity difference (IID) and inter-channel phase difference (IPD), play a fundamental role in recent deep learning based dual-microphone speech enhancement (DMSE) systems. However, learning the mutual relationship between artificially designed spatial and spectral features is hard in the end-to-end DMSE. In this work, a novel architecture for DMSE using a multi-head cross-attention based convolutional recurrent network (MHCA-CRN) is presented. The proposed MHCA-CRN model includes a channel-wise encoding structure for preserving intra-channel features and a multi-head cross-attention mechanism for fully exploiting cross-channel features. In addition, the proposed approach specifically formulates the decoder with an extra SNR estimator to estimate frame-level SNR under a multi-task learning framework, which is expected to avoid speech distortion led by end-to-end DMSE module. Finally, a spectral gain function is adopted to further suppress the unnatural residual noise. Experiment results demonstrated superior performance of the proposed model against several state-of-the-art models.
BibTeX
@inproceedings{xu2022improving,
  author = {Xinmeng Xu and Rongzhi Gu and Yuexian Zou},
  title = {{Improving Dual-Microphone Speech Enhancement by Learning Cross-Channel Features with Multi-Head Attention}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=dpfZ9_2ueXg},
}

Loading related content...

Figure