Abstract
In this article, we describe Conditioned Localizer and Classifier (CoLoC), a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes two stages: localization is performed first, followed by classification conditioned by the output of the localizer. To resolve the problem of an unknown number of sources, we incorporate Sequential Set Generation (SSG). Models from both stages are SELDnet-like Convolutional Recurrent Neural Networks (CRNNs) with single outputs. Our reasoning demonstrates that two single-output models are well-suited for the SELD task. We show that our solution improves upon the baseline system in most metrics on the STARSS22 Dataset.
Introduction
Sound Event Localization and Detection (SELD) is a complex task often referred to as the "cocktail party problem" for machines. It requires a system to simultaneously identify "what" a sound is (classification) and "where" it originates (localization). A significant challenge in this domain is homogeneous overlap—scenarios where multiple sources of the same class (e.g., two simultaneous speakers) occur at once.
Traditional approaches typically rely on Permutation Invariant Training (PIT) to handle these overlaps. However, PIT is computationally expensive, scaling at $N!$ relative to the number of sources, and can be difficult to optimize. CoLoC addresses these limitations by utilizing a decoupled, two-stage pipeline that avoids PIT entirely, favoring a recursive, location-first strategy.
Methodology
Two-Stage Pipeline
CoLoC splits the SELD task into two distinct, sequential components:
- Self-Conditioned SSG Localizer: Instead of predicting all source locations simultaneously, the model employs Sequential Set Generation (SSG). It predicts one Direction of Arrival (DOA) at a time. This predicted location is then fed back into the model as a condition, signaling the network to ignore that spatial coordinate and search for the next source. This recursive process continues until the model outputs a "stop" token, represented by the origin vector.
- Location-Conditioned Classifier: Following localization, a second model receives the audio features along with a specific predicted location. Its task is to determine the class of the sound event occurring at that exact spatial point.
Technical Implementation
Both stages utilize SELDnet-like CRNN architectures. For localization, the authors employ an $L_{1.5}$ norm loss function, which provides a balance between the mean-seeking $L_2$ and median-seeking $L_1$ losses. For classification, Focal Loss is used to mitigate the class imbalance caused by the high frequency of "silence" frames.
To ensure the classifier is robust to minor localization inaccuracies, the training process includes Angle Perturbation, where input angles are randomly shifted by up to 5 degrees. Additionally, a novel "Stacked-Tracks" data formatting method was developed to organize ground truth for recursive training, allowing the model to generalize to an arbitrary number of sources.
Experimental Setup
The models were evaluated using two primary datasets:
- Synthetic Dataset: 1,200 one-minute mixtures created from Spatial Room Impulse Responses (SRIRs) and FSD50K sound events.
- STARSS22: Real-world recordings (292 minutes) from Tokyo and Tampere, serving as the benchmark for the DCASE2022 Challenge Task 3.
Results and Discussion
CoLoC was compared against the DCASE2022 baseline using two configurations: max_ov2 and max_ov3 (referring to the maximum overlap handled).
| Metric | DCASE2022 Baseline | CoLoC (Best) |
|---|---|---|
| Localization Error ($LE_{CD}$) | 29.3° | 24.6° |
| F1-Score ($F_{20^\circ}$) | 21% | 33% |
| Localization Recall ($LR_{CD}$) | 46% | 51% |
| Error Rate ($ER_{20^\circ}$) | 0.71 | 0.76 |
CoLoC significantly outperformed the baseline in localization accuracy and F1-score. The improvement in $F_{20^\circ}$ (from 21% to 33%) highlights the effectiveness of using localization as a prior for classification. By focusing on a specific direction, the classifier's task is simplified.
However, the system showed a slightly higher Error Rate ($ER_{20^\circ}$) than the baseline. This is attributed to error propagation: as the localizer moves through recursive steps, its accuracy can degrade, occasionally leading to "hallucinated" events or false positives in later iterations.
Conclusion
CoLoC demonstrates that decoupling localization and classification into a recursive pipeline reduces computational complexity from $N!$ to $O(N)$ while improving performance on real-world datasets. While the system is susceptible to error propagation, the use of SSG and location-conditioned classification provides a robust framework for handling overlapping sound sources. Future work may explore hybrid approaches that combine recursive generation with PIT-based refinement to further reduce false positives.
Summary updated