PDPCRN: Parallel Dual-Path CRN with Bi-directional Inter-Branch Interactions for Multi-Channel Speech Enhancement

Jiahui Pan, Shulin He, Tianci Wu, Hui Zhang, Xueliang ZhangarXiv 2023 · OpenReview

Abstract

Multi-channel speech enhancement seeks to utilize spatial information to distinguish target speech from interfering signals. While deep learning approaches like the dual-path convolutional recurrent network (DPCRN) have made strides, challenges persist in effectively modeling inter-channel correlations and amalgamating multi-level information. In response, we introduce the Parallel Dual-Path Convolutional Recurrent Network (PDPCRN). This acoustic modeling architecture has two key innovations. First, a parallel design with separate branches extracts complementary features. Second, bi-directional modules enable cross-branch communication. Together, these facilitate diverse representation fusion and enhanced modeling. Experimental validation on TIMIT datasets underscores the prowess of PDPCRN. Notably, against baseline models like the standard DPCRN, PDPCRN not only outperforms in PESQ and STOI metrics but also boasts a leaner computational footprint with reduced parameters.

Introduction

Multi-channel speech enhancement (MCSE) aims to recover clean speech from noisy, multi-microphone recordings by leveraging spatial diversity—the subtle timing and volume differences between sensors. While existing deep learning models like the Dual-Path Convolutional Recurrent Network (DPCRN) have advanced the field, they often struggle to balance global temporal context with local spatial cues. Many current architectures treat multi-channel inputs as simple spectral data, failing to explicitly model the complex inter-channel correlations necessary for precise spatial filtering.

Proposed Method: The PDPCRN Architecture

The Parallel Dual-Path Convolutional Recurrent Network (PDPCRN) addresses these limitations through a specialized parallel processing pipeline and a novel bi-directional interaction mechanism.

Parallel Branch Design

Instead of a monolithic processing stream, PDPCRN splits the workload into two specialized branches:

  • Branch A (Temporal Modeling): Combines a Dual-Path RNN (DPRNN) with Self-Attention to focus on identifying salient speech components and modeling long-term temporal dependencies.
  • Branch B (Spatial Modeling): Integrates Depthwise Convolutions with a DPRNN. Depthwise convolutions are highly efficient at capturing spatial localization cues from multi-channel data without the parameter explosion typical of standard convolutions.

Bi-directional Interaction (Mixing Block)

The "secret sauce" of the architecture is the Mixing Block, which allows the two branches to exchange information rather than operating in isolation. This cross-pollination occurs in two directions:

  • Channel Interaction: Spatial features from Branch B are sent to Branch A to refine speech feature learning.
  • Spatial Interaction: Temporal context from Branch A is sent to Branch B to help the model better propagate spatial relationships.
Detailed architecture of the PDPCRN Mixing Block showing bi-directional interactions between Branch A (left) and Branch B (right).

Experimental Setup

The researchers evaluated PDPCRN using clean speech from the TIMIT database and noise from the DNS Challenge, NOISEX-92, and CHiME3 datasets. They simulated a 16-microphone uniform circular array in environments with reverberation times (RT60) ranging from 0.2s to 1.0s and Signal-to-Noise Ratios (SNR) from -10dB to 10dB.

Key Results and Discussion

PDPCRN consistently outperformed standard DPCRN and GCRN baselines across nearly all metrics while maintaining a smaller computational footprint.

  • Superior Quality and Intelligibility: PDPCRN achieved an average PESQ of 1.87 and an average STOI of 67.82%, compared to DPCRN's 1.80 and 65.61%, respectively.
  • Efficiency: Despite its parallel design, PDPCRN is leaner than the baseline, requiring only 790.78K parameters (vs. 814.60K for DPCRN) and fewer FLOPs (3.05G vs. 3.09G).
  • Ablation Study: The importance of the bi-directional interaction (BI) was confirmed through ablation testing. Removing the BI module resulted in a consistent performance drop across all SNR levels, highlighting that the branches must "inform" each other to reach peak accuracy.
Performance comparison of PDPCRN with and without Bi-directional Interaction (BI) across various SNR levels.

Conclusion

PDPCRN represents a significant step forward in multi-channel speech enhancement by specializing in both "what" is being said and "where" it is coming from. By allowing temporal and spatial branches to interact bi-directionally, the model achieves state-of-the-art results with improved efficiency. Future work will explore the model's performance on smaller microphone arrays and real-world acoustic anomalies.

Summary updated

Abstract

Multi-channel speech enhancement seeks to utilize spatial information to distinguish target speech from interfering signals. While deep learning approaches like the dual-path convolutional recurrent network (DPCRN) have made strides, challenges persist in effectively modeling inter-channel correlations and amalgamating multi-level information. In response, we introduce the Parallel Dual-Path Convolutional Recurrent Network (PDPCRN). This acoustic modeling architecture has two key innovations. First, a parallel design with separate branches extracts complementary features. Second, bi-directional modules enable cross-branch communication. Together, these facilitate diverse representation fusion and enhanced modeling. Experimental validation on TIMIT datasets underscores the prowess of PDPCRN. Notably, against baseline models like the standard DPCRN, PDPCRN not only outperforms in PESQ and STOI metrics but also boasts a leaner computational footprint with reduced parameters.
BibTeX
@misc{pan2023pdpcrn,
  author = {Jiahui Pan and Shulin He and Tianci Wu and Hui Zhang and Xueliang Zhang},
  title = {{PDPCRN: Parallel Dual-Path CRN with Bi-directional Inter-Branch Interactions for Multi-Channel Speech Enhancement}},
  year = {2023},
  eprint = {2309.10379},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.10379},
}
Figure