TPARN: Triple-Path Attentive Recurrent Network for Time-Domain Multichannel Speech Enhancement

Ashutosh Pandey, Buye Xu, Anurag Kumar, Jacob Donley, Paul Calamia, DeLiang Wang

TL;DR

TPARN (Triple-path Attentive Recurrent Network) is a novel end-to-end architecture that brings "triple-path" processing to multichannel speech enhancement. By adding a spatial processing dimension to existing dual-path temporal models, it achieves state-of-the-art performance in removing both noise and reverberation directly in the time domain.

What Problem Does This Paper Address?

Speech enhancement aims to clean up audio degraded by background noise and room reverberation. While single-channel (monaural) methods have improved significantly with deep learning, they are limited to time-frequency information. Multichannel setups (using microphone arrays) offer a "spatial" advantage, but effectively fusing temporal and spatial information remains a challenge.

Most existing multichannel models operate in the frequency domain (using STFT) or rely on traditional beamforming techniques. Time-domain models, which map noisy samples directly to clean ones, have shown promise for denoising but have historically struggled with reverberation—the "echoey" effect of a room. The authors identify a gap: the need for an end-to-end time-domain model that can explicitly and effectively model spatial relationships across multiple microphones.

Methods

The core innovation is the Triple-path Attentive Recurrent Network (TPARN). It builds upon the concept of dual-path networks (which split audio into local "chunks" to handle short-term and long-term dependencies) and adds a third dimension for spatial processing.

The Triple-Path Architecture

The model processes audio in three distinct stages:

  1. Intra-chunk processing: Models local temporal patterns within short segments of audio.
  2. Inter-chunk processing: Models global temporal patterns across the entire utterance.
  3. Inter-channel processing: The "third path" that aggregates spatial context across all microphones.

The ARN Block

Each path utilizes an Attentive Recurrent Network (ARN). This is a hybrid block consisting of:

  • RNN (BLSTM): To capture sequential dependencies.
  • Self-Attention: To capture long-range relationships that RNNs might miss.
  • Feedforward Network: For non-linear feature transformation.

MIMO Capability

Unlike many systems that take multiple inputs but produce only one "best" output (MISO), TPARN is a Multiple-Input Multiple-Output (MIMO) architecture. It enhances all input channels simultaneously, which helps preserve spatial cues for any downstream processing.

Loss Function

The researchers used the Phase Constrained Magnitude (PCM) loss. Unlike standard Mean Squared Error (MSE), PCM loss is specifically designed to handle the artifacts that often occur when training time-domain networks with magnitude-based objectives.

Datasets

The model was evaluated on two datasets:

  1. WSJCAM0 + REVERB Challenge: A standard dataset using a 4-microphone circular array with simulated reverberation (T60 between 0.2 to 1.2 seconds).
  2. Spatialized DNS Challenge 2020: A significantly more difficult dataset. The authors used the DNS Challenge corpus and "spatialized" it using Pyroomacoustics, simulating complex room geometries, varying noise sources, and diverse signal-to-noise ratios (SNR).

Key Insights and Ideas

  • Spatial Modeling Matters: The addition of the third "inter-channel" path is what allows the model to outperform single-channel and standard dual-path models.
  • The Power of Hybridization: Using ARN (RNN + Attention) for spatial modeling proved superior to using just an RNN or just an Attention mechanism. The RNN handles the sequence, while Attention helps the model "focus" on the most relevant spatial features.
  • MIMO vs. MISO: The authors found that TPARN-MIMO (enhancing all channels) performed slightly better than TPARN-MISO (enhancing just one). This suggests that learning to reconstruct the entire spatial field helps the model learn better internal representations of the speech.
  • Time-Domain Reverberation Success: TPARN demonstrates that time-domain models can handle heavy reverberation effectively if the architecture is deep enough and captures spatial context correctly.

Key Results

TPARN set new benchmarks across several metrics, including SI-SDR (Signal-to-Distortion Ratio), STOI (Intelligibility), and PESQ (Quality).

  • On the difficult DNS dataset: TPARN achieved an SI-SDR of 8.4 dB, a massive improvement over the second-best baseline, DCRN-MISO, which reached 4.6 dB.
  • PESQ Gains: On the same dataset, TPARN reached a PESQ of 2.75, compared to 2.57 for the best baseline.
  • Loss Function Impact: Switching from MSE to PCM loss improved the PESQ score from 3.07 to 3.42 on the WSJCAM0 dataset, proving the importance of phase-aware training.

Limitations and Future Work

While TPARN is highly effective, the authors note a few areas for growth:

  • Fixed Geometry: The current experiments were conducted on a fixed circular array. The performance on "ad-hoc" arrays (microphones placed randomly) is yet to be fully explored.
  • Moving Sources: The datasets used primarily featured stationary speakers. Real-world scenarios often involve moving sources, which adds a layer of spatial complexity.
  • Computational Load: Triple-path processing with self-attention is computationally intensive, which may pose challenges for real-time implementation on low-power edge devices.

Summary updated

Abstract

In this work, we propose a new model called triple-path attentive recurrent network (TPARN) for multichannel speech enhancement in the time domain. TPARN extends a single-channel dual-path network to a multichannel network by adding a third path along the spatial dimension. First, TPARN processes speech signals from all channels independently using a dual-path attentive recurrent network (ARN), which is a recurrent neural network (RNN) augmented with self-attention. Next, an ARN is introduced along the spatial dimension for spatial context aggregation. TPARN is designed as a multiple-input and multiple-output architecture to enhance all input channels simultaneously. Experimental results demonstrate the superiority of TPARN over existing state-of-the-art approaches.
BibTeX
@inproceedings{pandey2022tparn,
  author = {Ashutosh Pandey and Buye Xu and Anurag Kumar and Jacob Donley and Paul Calamia and DeLiang Wang},
  title = {{TPARN: Triple-Path Attentive Recurrent Network for Time-Domain Multichannel Speech Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=HrcFanNNSJ},
}

Loading related content...

Figure