TPARN: Triple-path Attentive Recurrent Network for Time-domain Multichannel Speech Enhancement

Ashutosh Pandey, Buye Xu, Anurag Kumar, Jacob Donley, Paul Calamia, DeLiang Wang

Abstract

In this work, we propose a new model called triple-path attentive recurrent network (TPARN) for multichannel speech enhancement in the time domain. TPARN extends a single-channel dual-path network to a multichannel network by adding a third path along the spatial dimension. First, TPARN processes speech signals from all channels independently using a dual-path attentive recurrent network (ARN), which is a recurrent neural network (RNN) augmented with self-attention. Next, an ARN is introduced along the spatial dimension for spatial context aggregation. TPARN is designed as a multiple-input and multiple-output architecture to enhance all input channels simultaneously. Experimental results demonstrate the superiority of TPARN over existing state-of-the-art approaches.

Introduction

Speech enhancement is a critical task aimed at cleaning up audio signals degraded by background noise and room reverberation. While single-channel (monaural) methods have seen significant improvements through deep learning, they are inherently limited to time-frequency information. Multichannel setups, utilizing microphone arrays, offer a distinct "spatial" advantage. However, effectively fusing temporal and spatial information remains a significant challenge in the field.

Most existing multichannel models operate in the frequency domain using Short-Time Fourier Transforms (STFT) or rely on traditional beamforming techniques. While time-domain models have shown promise for denoising, they have historically struggled with reverberation—the complex "echoey" effect caused by room acoustics. TPARN addresses this gap by providing an end-to-end time-domain model that explicitly and effectively models spatial relationships across multiple microphones.

The Triple-Path Architecture

The core innovation of this work is the Triple-path Attentive Recurrent Network (TPARN). It builds upon the concept of dual-path networks, which split audio into local "chunks" to handle short-term and long-term dependencies, and introduces a third dimension specifically for spatial processing.

The TPARN block architecture illustrating the sequential processing of intra-chunk, inter-chunk, and inter-channel paths.

The model processes audio in three distinct stages:

  1. Intra-chunk processing: Models local temporal patterns within short segments of audio.
  2. Inter-chunk processing: Models global temporal patterns across the entire utterance.
  3. Inter-channel processing: The "third path" that aggregates spatial context across all microphones.

Each path utilizes an Attentive Recurrent Network (ARN) block. This hybrid architecture combines a Bidirectional Long Short-Term Memory (BLSTM) network to capture sequential dependencies with a self-attention mechanism to capture long-range relationships. A feedforward network is then used for non-linear feature transformation.

End-to-End System Design

TPARN is designed as a deep, end-to-end architecture. The system transforms raw multichannel audio into frames and then into chunks before passing them through a series of TPARN blocks.

The overall end-to-end TPARN system pipeline, from raw multichannel audio input to enhanced output.

Unlike many systems that take multiple inputs but produce only one "best" output (MISO), TPARN is a Multiple-Input Multiple-Output (MIMO) architecture. It enhances all input channels simultaneously, which helps preserve spatial cues for any downstream processing. To train this complex architecture effectively, the researchers employed the Phase Constrained Magnitude (PCM) loss. Unlike standard Mean Squared Error (MSE), PCM loss is specifically designed to handle artifacts that often occur when training time-domain networks with magnitude-based objectives.

Experimental Results

The model was evaluated on two rigorous datasets: the WSJCAM0 + REVERB Challenge and the Spatialized DNS Challenge 2020. TPARN set new benchmarks across several key metrics:

  • Spatial Modeling Success: The addition of the third "inter-channel" path allowed the model to significantly outperform single-channel and standard dual-path models.
  • DNS Dataset Performance: TPARN achieved an SI-SDR of 8.4 dB, a massive improvement over the second-best baseline, DCRN-MISO (4.6 dB). It also reached a PESQ of 2.75.
  • Loss Function Impact: Switching from MSE to PCM loss improved the PESQ score from 3.07 to 3.42 on the WSJCAM0 dataset, validating the importance of phase-aware training.

Conclusion

TPARN demonstrates that time-domain models can effectively handle heavy reverberation and noise by utilizing a triple-path architecture that captures both temporal and spatial contexts. By hybridizing RNNs with self-attention and employing a MIMO design, TPARN achieves state-of-the-art performance in multichannel speech enhancement. Future work will explore performance on ad-hoc microphone arrays and moving sound sources.

Summary updated

Abstract

In this work, we propose a new model called triple-path attentive recurrent network (TPARN) for multichannel speech enhancement in the time domain. TPARN extends a single-channel dual-path network to a multichannel network by adding a third path along the spatial dimension. First, TPARN processes speech signals from all channels independently using a dual-path attentive recurrent network (ARN), which is a recurrent neural network (RNN) augmented with self-attention. Next, an ARN is introduced along the spatial dimension for spatial context aggregation. TPARN is designed as a multiple-input and multiple-output architecture to enhance all input channels simultaneously. Experimental results demonstrate the superiority of TPARN over existing state-of-the-art approaches.
BibTeX
@inproceedings{pandey2022tparn,
  author = {Ashutosh Pandey and Buye Xu and Anurag Kumar and Jacob Donley and Paul Calamia and DeLiang Wang},
  title = {{TPARN: Triple-path Attentive Recurrent Network for Time-domain Multichannel Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=HrcFanNNSJ},
}
Figure