Mask-Weighted Spatial Likelihood Coding for Speaker-Independent Joint Localization and Mask Estimation

Jakob Kienegger, Alina Mannanova, Timo Gerkmann

Abstract

Jointly locating and separating multiple sound sources in a "cocktail party" environment remains a significant challenge for speech processing systems. This paper presents Mask-Weighted Spatial Likelihood Coding (MW-SLC), a novel encoding scheme that enables a single deep neural network to perform simultaneous Direction of Arrival (DoA) estimation and Time-Frequency (T-F) mask estimation. By replacing sparse binary spatial labels with "blurred" Gaussian likelihoods, MW-SLC mitigates the vanishing gradient problem that typically plagues high-resolution spatial grids. Experimental results on the MC-LibriMix dataset demonstrate that MW-SLC significantly outperforms binary baselines, achieving high localization precision and separation quality close to oracle performance.

Introduction

In multi-channel speech separation, systems must identify both the spatial origin (DoA) of a speaker and the specific T-F regions of the audio spectrum belonging to that speaker (the mask). Traditional speaker-independent systems often employ a spatial grid to discretize the environment. However, increasing the grid's resolution (e.g., to 1° slices) leads to extreme label sparsity. In a binary coding scheme, where only the active speaker's bin is marked with a '1', the overwhelming majority of zero-valued bins causes gradients to vanish during training. This often results in models that fail to learn, defaulting to "zero everywhere" predictions.

Proposed Methodology: MW-SLC

To address the training bottleneck of binary coding, we propose moving to Likelihood Coding. Instead of a discrete point, the target for each speaker is represented by a Gaussian curve centered at their true DoA.

1. Encoding Scheme

The height of this Gaussian curve is scaled by the speaker’s Ideal Ratio Mask (IRM). This "blurring" creates a continuous optimization landscape (a "slope") for the neural network to follow, preventing gradient collapse. Mathematically, the paper provides a formal proof showing that while the gradient for binary coding approaches zero as grid partitions increase, the gradient for MW-SLC remains robust and positive.

2. System Architecture and Inference

The MW-SLC system architecture for joint localization and mask estimation. Source, p. 3

The MW-SLC system architecture for joint localization and mask estimation.

As shown in Figure 1, the DNN outputs a spatial likelihood grid $\hat{L}_{tk\theta}$. During inference, the system performs a peak search on the time-averaged likelihood $\hat{\ell}_{t\theta}$ to identify speaker DoAs. The grid is then sampled at these identified peaks to extract the corresponding clean speech masks $\hat{M}_{tk}^{(i)}$. This approach allows a separation model to act as its own Sound Source Localization (SSL) system.

Experimental Evaluation

The proposed method was evaluated using the MC-LibriMix dataset, featuring simulated 4-channel linear microphone arrays with reverberation times (RT60) between 200ms and 600ms. Models were trained on two-speaker mixtures (MC-Libri2Mix) and evaluated on both two and three-speaker mixtures (MC-Libri3Mix) to test speaker-independent generalization.

Key Results

The transition from binary coding (MW-SBC) to the proposed MW-SLC yielded dramatic improvements across all metrics.

Quantitative comparison of MW-SLC against binary baselines and oracle models.

As detailed in Table 1 (Figure 4), the Mean Absolute Error (MAE) for localization dropped from 20.74° with binary coding to just 1.26° with MW-SLC. Reliability also saw a significant boost, with precision and recall for speaker detection jumping from approximately 50% to over 90%. In terms of separation, the system achieved a Signal-to-Distortion Ratio improvement ($\Delta$SI-SDR) of 5.03 dB, nearing the performance of separation-only models where locations are known a priori (e.g., 5.85 dB for MC-CRUSE).

Discussion and Conclusion

The "blurring" advantage of MW-SLC effectively solves the label imbalance problem by increasing the "active" area of training data. This prevents the model from collapsing into an all-zero output. Furthermore, the model successfully generalized to three-speaker scenarios despite being trained only on two, confirming its speaker-independent nature.

Future work will focus on refining spectral details, as the current likelihood approach can occasionally "smear" harmonic structures. Incorporating a secondary loss function for mask reconstruction and testing on real-world, non-simulated acoustic environments are key next steps for this technology.

Summary updated

BibTeX
@inproceedings{kienegger2025mask,
  author = {Jakob Kienegger and Alina Mannanova and Timo Gerkmann},
  title = {{Mask-Weighted Spatial Likelihood Coding for Speaker-Independent Joint Localization and Mask Estimation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10887963},
}
Figure