A dual-stream deep attractor network with multi-domain learning for speech dereverberation and separation

Hangting Chen, Pengyuan Zhang

Abstract

Speech separation in reverberant environments remains a significant challenge due to the overlap of direct sound and late reflections. We present the Time-Domain Deep Attractor Network (TD-DAN), a two-stream architecture designed to simultaneously perform speaker separation and dereverberation. By decoupling speaker identification from waveform reconstruction and targeting early reflections rather than anechoic signals, TD-DAN achieves state-of-the-art performance on the SMS-WSJ dataset, outperforming established models like Conv-TasNet.

Introduction

Real-world speech processing must contend with multiple concurrent speakers and acoustic reverberation. While the Deep Attractor Network (DAN) effectively separates speakers in clean conditions by mapping signals to an embedding space, its performance degrades in reverberant settings where the assumption of time-frequency bin sparsity fails. Existing methods often treat separation and dereverberation as sequential tasks. This paper introduces TD-DAN, a unified framework that addresses both challenges within a single time-domain architecture.

Methodology: The Two-Stream Architecture

The core of TD-DAN is its dual-stream design, which separates the tasks of identifying speakers and extracting their specific waveforms.

The proposed TD-DAN architecture featuring the Speaker Encoding Stream (SES) and Speech Decoding Stream (SDS).
  1. Speaker Encoding Stream (SES): This stream acts as the "identity finder." It processes the mixture to create a high-dimensional embedding space where "attractors" (centroids representing individual speakers) are identified. The SES can utilize various encoders, such as STFT or learnable convolutional kernels, allowing for multi-domain learning.
  2. Speech Decoding Stream (SDS): Functioning as the "extractor," this stream uses the attractors generated by the SES to filter the original mixture and reconstruct the time-domain waveforms for each speaker.

Key Innovations and Loss Functions

To handle the "smearing" effect of reverberation, TD-DAN incorporates specialized loss functions and learning targets.

Visualization of the embedding space and the auxiliary clustering losses (Concentration and Discrimination).
  • Auxiliary Clustering Losses: The model employs a Concentration Loss to pull T-F bin embeddings closer to their respective attractors and a Discrimination Loss to maximize the distance between different speakers' attractors. This counters the overlap caused by echoes.
  • Early Reflection Target: Rather than aiming for a perfectly dry (anechoic) signal, TD-DAN targets "early reflections." These initial sound bounces preserve intelligibility while avoiding the chaotic complexity of late reflections, making the learning objective more attainable and effective for downstream tasks like speech recognition.

Results and Discussion

Evaluations on the SMS-WSJ dataset demonstrate the superiority of the TD-DAN approach. In 2-speaker mixtures, TD-DAN achieved a Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) gain of 9.79 dB, surpassing Conv-TasNet by 0.91 dB and the original DAN by 1.92 dB. Performance remained robust in 3-speaker scenarios, maintaining a 0.47 dB lead over Conv-TasNet. Crucially, the model also yielded the lowest Word Error Rates (WER), indicating that the dereverberation process significantly enhances speech intelligibility for automated systems.

Conclusion

TD-DAN represents a significant advancement in monaural speech separation for reverberant environments. By leveraging a two-stream architecture and targeting early reflections, it effectively mitigates the impact of late reflections. Future work will explore the integration of multi-channel spatial information to further improve performance in complex acoustic scenes.

Summary updated

BibTeX
@article{chen2021dual,
  author = {Hangting Chen and Pengyuan Zhang},
  title = {{A dual-stream deep attractor network with multi-domain learning for speech dereverberation and separation}},
  journal = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2021.04.023},
}
Figure