Abstract
Jointly locating and separating multiple sound sources in a "cocktail party" environment remains a significant challenge for speech processing systems. This paper presents Mask-Weighted Spatial Likelihood Coding (MW-SLC), a novel encoding scheme that enables a single deep neural network to perform simultaneous Direction of Arrival (DoA) estimation and Time-Frequency (T-F) mask estimation. By replacing sparse binary spatial labels with "blurred" Gaussian likelihoods, MW-SLC mitigates the vanishing gradient problem that typically plagues high-resolution spatial grids. Experimental results on the MC-LibriMix dataset demonstrate that MW-SLC significantly outperforms binary baselines, achieving high localization precision and separation quality close to oracle performance.
Introduction
In multi-channel speech separation, systems must identify both the spatial origin (DoA) of a speaker and the specific T-F regions of the audio spectrum belonging to that speaker (the mask). Traditional speaker-independent systems often employ a spatial grid to discretize the environment. However, increasing the grid's resolution (e.g., to 1° slices) leads to extreme label sparsity. In a binary coding scheme, where only the active speaker's bin is marked with a '1', the overwhelming majority of zero-valued bins causes gradients to vanish during training. This often results in models that fail to learn, defaulting to "zero everywhere" predictions.
Proposed Methodology: MW-SLC
To address the training bottleneck of binary coding, we propose moving to Likelihood Coding. Instead of a discrete point, the target for each speaker is represented by a Gaussian curve centered at their true DoA.
1. Encoding Scheme
The height of this Gaussian curve is scaled by the speaker’s Ideal Ratio Mask (IRM). This "blurring" creates a continuous optimization landscape (a "slope") for the neural network to follow, preventing gradient collapse. Mathematically, the paper provides a formal proof showing that while the gradient for binary coding approaches zero as grid partitions increase, the gradient for MW-SLC remains robust and positive.
2. System Architecture and Inference
The MW-SLC system architecture for joint localization and mask estimation. Source, p. 3
As shown in Figure 1, the DNN outputs a spatial likelihood grid $\hat{L}_{tk\theta}$. During inference, the system performs a peak search on the time-averaged likelihood $\hat{\ell}_{t\theta}$ to identify speaker DoAs. The grid is then sampled at these identified peaks to extract the corresponding clean speech masks $\hat{M}_{tk}^{(i)}$. This approach allows a separation model to act as its own Sound Source Localization (SSL) system.
Experimental Evaluation
The proposed method was evaluated using the MC-LibriMix dataset, featuring simulated 4-channel linear microphone arrays with reverberation times (RT60) between 200ms and 600ms. Models were trained on two-speaker mixtures (MC-Libri2Mix) and evaluated on both two and three-speaker mixtures (MC-Libri3Mix) to test speaker-independent generalization.
Key Results
The transition from binary coding (MW-SBC) to the proposed MW-SLC yielded dramatic improvements across all metrics.
As detailed in Table 1 (Figure 4), the Mean Absolute Error (MAE) for localization dropped from 20.74° with binary coding to just 1.26° with MW-SLC. Reliability also saw a significant boost, with precision and recall for speaker detection jumping from approximately 50% to over 90%. In terms of separation, the system achieved a Signal-to-Distortion Ratio improvement ($\Delta$SI-SDR) of 5.03 dB, nearing the performance of separation-only models where locations are known a priori (e.g., 5.85 dB for MC-CRUSE).
Discussion and Conclusion
The "blurring" advantage of MW-SLC effectively solves the label imbalance problem by increasing the "active" area of training data. This prevents the model from collapsing into an all-zero output. Furthermore, the model successfully generalized to three-speaker scenarios despite being trained only on two, confirming its speaker-independent nature.
Future work will focus on refining spectral details, as the current likelihood approach can occasionally "smear" harmonic structures. Incorporating a secondary loss function for mask reconstruction and testing on real-world, non-simulated acoustic environments are key next steps for this technology.
Summary updated