Exploring the time-domain deep attractor network with two-stream architectures in a reverberant environment

Hangting Chen, Pengyuan Zhang

Abstract

This paper introduces the Time-Domain Deep Attractor Network (TD-DAN), a novel architecture designed for simultaneous multi-speaker separation and dereverberation in complex acoustic environments. By integrating the attractor paradigm into the time domain via a two-stream approach—comprising a Speaker Encoding Stream (SES) and a Speech Decoding Stream (SDS)—the model overcomes the limitations of traditional frequency-domain networks and permutation-invariant training. Evaluated on the SMS-WSJ dataset, TD-DAN demonstrates superior performance over industry standards like Conv-TasNet, particularly in reverberant conditions, by effectively targeting early reflections and utilizing auxiliary clustering losses.

Introduction: The Challenge of Reverberant Environments

In the "cocktail party problem," the goal is to isolate a single voice from a mixture of multiple speakers. While significant progress has been made for "clean" recordings, real-world scenarios are often plagued by reverberation—sound reflections from surfaces that distort the signal.

Existing Deep Attractor Networks (DANs) typically operate in the Time-Frequency (T-F) domain using spectrograms. However, spectrograms have fixed resolutions and often struggle with phase reconstruction. Conversely, time-domain models like Conv-TasNet are powerful but rely on Permutation Invariant Training (PIT), which can be less effective with a variable number of speakers or complex echoes. TD-DAN aims to bridge this gap by combining the discriminative power of speaker attractors with high-fidelity time-domain convolutional processing.

The TD-DAN Architecture

The core of the TD-DAN is its two-stream architecture, which separates the tasks of identifying speakers and extracting their signals.

The two-stream architecture of TD-DAN, featuring the Speaker Encoding Stream (SES) and Speech Decoding Stream (SDS).
  1. Speaker Encoding Stream (SES): This branch acts as the "identity seeker." It processes the mixture to create a high-dimensional embedding space where each point represents a specific time-frequency bin. It then calculates "attractors" — centroids in this space that represent individual speakers.
  2. Speech Decoding Stream (SDS): This branch is the "signal extractor." It uses the attractors from the SES as a guide to isolate the target speaker's voice from the raw audio.

Unlike original DANs that use Short-Time Fourier Transform (STFT), TD-DAN employs learnable "free" encoders or stacked STFT kernels that operate directly on the waveform, preserving more signal information.

Key Innovations and Learning Strategies

A major challenge in dereverberation is choosing the right target. The authors found that aiming for a perfectly "dry" (anechoic) signal is often too difficult for the network to learn. Instead, they target early reflections (the first 50ms of echoes), which preserves speech intelligibility while providing a more stable learning objective.

Furthermore, to enhance the distinctness of speaker embeddings, the model incorporates auxiliary clustering losses.

Visualization of the attractor space and auxiliary clustering losses.

As shown in the figure above, concentration losses pull embedding points toward their respective speaker's attractor, while discrimination losses push the attractors of different speakers further apart. These losses narrow the performance gap between training (with oracle labels) and testing (using K-means clustering).

Experimental Results

The model was rigorously tested on the Spatialized Multi-Speaker Wall Street Journal (SMS-WSJ) dataset, which simulates realistic room acoustics with varying sizes and reverberation times (T60 from 200ms to 500ms).

  • Performance: On the 2-speaker reverberant test set, TD-DAN achieved a 9.79 dB SI-SDR (Scale-Invariant Source-to-Distortion Ratio).
  • Comparison: This represents a 1.92 dB improvement over the baseline DAN and a 0.91 dB improvement over Conv-TasNet.
  • Flexibility: TD-DAN generalizes well to a variable number of speakers. A single model trained on 1, 2, and 3-speaker mixtures outperformed specialized models, highlighting the flexibility of the attractor-based approach.

Conclusion and Future Work

TD-DAN successfully integrates time-domain processing with the attractor paradigm to tackle speaker separation and dereverberation. While the two-stream architecture introduces some computational overhead and still relies on K-means clustering for inference, its performance gains in reverberant environments are significant. Future research may explore multi-channel implementations to further enhance dereverberation capabilities using microphone arrays.

Summary updated

BibTeX
@article{chen2021exploring,
  author = {Hangting Chen and Pengyuan Zhang},
  title = {{Exploring the time-domain deep attractor network with two-stream architectures in a reverberant environment}},
  journal = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2021.04.023},
}
Figure