CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection

Slawomir Kapka, Jakub Tkaczuk

Abstract

In this article, we describe Conditioned Localizer and Classifier (CoLoC), a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes two stages: localization is performed first, followed by classification conditioned by the output of the localizer. To resolve the problem of an unknown number of sources, we incorporate Sequential Set Generation (SSG). Models from both stages are SELDnet-like Convolutional Recurrent Neural Networks (CRNNs) with single outputs. Our reasoning demonstrates that two single-output models are well-suited for the SELD task. We show that our solution improves upon the baseline system in most metrics on the STARSS22 Dataset.

Introduction

Sound Event Localization and Detection (SELD) is a complex task often referred to as the "cocktail party problem" for machines. It requires a system to simultaneously identify "what" a sound is (classification) and "where" it originates (localization). A significant challenge in this domain is homogeneous overlap—scenarios where multiple sources of the same class (e.g., two simultaneous speakers) occur at once.

Traditional approaches typically rely on Permutation Invariant Training (PIT) to handle these overlaps. However, PIT is computationally expensive, scaling at $N!$ relative to the number of sources, and can be difficult to optimize. CoLoC addresses these limitations by utilizing a decoupled, two-stage pipeline that avoids PIT entirely, favoring a recursive, location-first strategy.

Methodology

Two-Stage Pipeline

CoLoC splits the SELD task into two distinct, sequential components:

  1. Self-Conditioned SSG Localizer: Instead of predicting all source locations simultaneously, the model employs Sequential Set Generation (SSG). It predicts one Direction of Arrival (DOA) at a time. This predicted location is then fed back into the model as a condition, signaling the network to ignore that spatial coordinate and search for the next source. This recursive process continues until the model outputs a "stop" token, represented by the origin vector.
  2. Location-Conditioned Classifier: Following localization, a second model receives the audio features along with a specific predicted location. Its task is to determine the class of the sound event occurring at that exact spatial point.

Technical Implementation

Both stages utilize SELDnet-like CRNN architectures. For localization, the authors employ an $L_{1.5}$ norm loss function, which provides a balance between the mean-seeking $L_2$ and median-seeking $L_1$ losses. For classification, Focal Loss is used to mitigate the class imbalance caused by the high frequency of "silence" frames.

To ensure the classifier is robust to minor localization inaccuracies, the training process includes Angle Perturbation, where input angles are randomly shifted by up to 5 degrees. Additionally, a novel "Stacked-Tracks" data formatting method was developed to organize ground truth for recursive training, allowing the model to generalize to an arbitrary number of sources.

Experimental Setup

The models were evaluated using two primary datasets:

  • Synthetic Dataset: 1,200 one-minute mixtures created from Spatial Room Impulse Responses (SRIRs) and FSD50K sound events.
  • STARSS22: Real-world recordings (292 minutes) from Tokyo and Tampere, serving as the benchmark for the DCASE2022 Challenge Task 3.

Results and Discussion

CoLoC was compared against the DCASE2022 baseline using two configurations: max_ov2 and max_ov3 (referring to the maximum overlap handled).

Metric DCASE2022 Baseline CoLoC (Best)
Localization Error ($LE_{CD}$) 29.3° 24.6°
F1-Score ($F_{20^\circ}$) 21% 33%
Localization Recall ($LR_{CD}$) 46% 51%
Error Rate ($ER_{20^\circ}$) 0.71 0.76

CoLoC significantly outperformed the baseline in localization accuracy and F1-score. The improvement in $F_{20^\circ}$ (from 21% to 33%) highlights the effectiveness of using localization as a prior for classification. By focusing on a specific direction, the classifier's task is simplified.

However, the system showed a slightly higher Error Rate ($ER_{20^\circ}$) than the baseline. This is attributed to error propagation: as the localizer moves through recursive steps, its accuracy can degrade, occasionally leading to "hallucinated" events or false positives in later iterations.

Conclusion

CoLoC demonstrates that decoupling localization and classification into a recursive pipeline reduces computational complexity from $N!$ to $O(N)$ while improving performance on real-world datasets. While the system is susceptible to error propagation, the use of SSG and location-conditioned classification provides a robust framework for handling overlapping sound sources. Future work may explore hybrid approaches that combine recursive generation with PIT-based refinement to further reduce false positives.

Summary updated

Abstract

In this article, we describe Conditioned Localizer and Classifier (CoLoC) which is a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes of two stages: the localization is done first and is followed by classification conditioned by the output of the localizer. In order to resolve the problem of the unknown number of sources we incorporate the idea borrowed from Sequential Set Generation (SSG). Models from both stages are SELDnet-like CRNNs, but with single outputs. Conducted reasoning shows that such two single-output models are fit for SELD task. We show that our solution improves on the baseline system in most metrics on the STARSS22 Dataset.
BibTeX
@inproceedings{kapka2022coloc,
  author = {Slawomir Kapka and Jakub Tkaczuk},
  title = {{CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=H5AWGZWACAW},
}
Figure