Abstract
Solving the "cocktail party problem" remains a significant challenge in machine hearing, particularly when the number of speakers is unknown. We present SDNet, a dual-channel speech separation network that sequentially infers speaker identity and spatial direction. By treating these as sequential cues, SDNet can dynamically separate an arbitrary number of overlapping voices with high precision. Experimental results on the WSJ0 benchmarks demonstrate that SDNet achieves state-of-the-art performance in both anechoic and reverberant environments, significantly outperforming existing single-channel and dual-channel baselines.
Introduction
In complex acoustic environments, humans effortlessly focus on a single target voice amidst background noise and competing talkers. Replicating this ability in AI models involves overcoming three primary hurdles: determining an unknown number of sources, solving the permutation problem (consistent speaker assignment across segments), and selecting the target stream. While single-channel methods have advanced, they lack the spatial depth inherent in binaural hearing. SDNet bridges this gap by leveraging both spatial and spectral features to identify and extract voices sequentially.
Methods
SDNet is composed of three integrated modules designed to learn spatial relationships and speaker characteristics end-to-end.
Feature Extraction Module
Unlike traditional signal processing that relies on hand-crafted features like Inter-channel Phase Difference (IPD), SDNet utilizes a 1D Convolutional Encoder paired with an Inter-channel Attention Correlation (IAC) mechanism. This allows the network to automatically learn the spatial relationships between microphones directly from raw waveforms, enhancing robustness.
Inference Module
The system's "brain" employs a Sequence-to-Sequence (Seq2Seq) approach with Attentive Decoders. It sequentially predicts a speaker's identity and their spatial direction (quantized into 37 labels at 5-degree intervals). The use of an <EOS> (End-of-Sequence) token allows the model to dynamically count the number of speakers in the mixture.
Separation Module
The inferred speaker and direction information serve as a "mask" for a Temporal Convolutional Network (TCN). This module, inspired by the Conv-TasNet architecture, extracts the specific audio stream for each identified speaker. The entire system is trained using a multi-task loss function that optimizes both separation quality (SI-SNR) and identification accuracy.
Experimental Setup
The model was evaluated using the WSJ0 (Wall Street Journal) speech separation benchmarks, including:
- WSJ0-2mix & WSJ0-3mix: Fixed two- and three-speaker mixtures.
- WSJ0-2&3mix: Variable mixtures to test source counting capabilities.
- Acoustic Conditions: Tests were conducted in both anechoic (clean) and reverberant (echoey) simulated environments to assess real-world applicability.
Key Results
SDNet demonstrates significant improvements in Signal-to-Distortion Ratio (SDR) and source counting accuracy.
As shown in the results table, SDNet achieved an SDR improvement of 25.31 dB on the anechoic WSJ0-2mix dataset, surpassing the dual-channel TasNet baseline (25.08 dB). In the more complex WSJ0-2&3mix scenario under reverberant conditions, SDNet maintained a high level of performance, correctly identifying the number of speakers 89.73% of the time—a substantial improvement over baseline models which typically range between 81% and 85%.
The sequential parsing strategy effectively eliminates the "permutation problem" by identifying the speaker before separation. Furthermore, the fusion of spatial and spectral information proves critical; knowing a sound's origin makes identifying the speaker easier, and vice versa.
Conclusion and Future Work
SDNet represents a significant step forward in multi-channel speech separation by integrating spatial inference with sequential source parsing. While the model is robust to reverberation, echoey environments still impact direction inference accuracy more than anechoic ones. Future research will explore continuous direction estimation to accommodate moving speakers and investigate methods to reduce computational complexity for real-time deployment on edge devices.
Summary updated