CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection

Slawomir Kapka, Jakub Tkaczuk

Introduction

Sound Event Localization and Detection (SELD) represents a significant challenge in machine listening, requiring systems to simultaneously identify sound categories and their precise 3D spatial coordinates. A persistent bottleneck in this field is "homogeneous overlap"—the occurrence of multiple simultaneous events belonging to the same class, such as two people speaking at once. Traditional models like SELDNet often fail to distinguish these as separate entities, while modern solutions relying on Permutation Invariant Training (PIT) introduce significant computational complexity.

To address these issues, researchers from Samsung R&D Institute Poland introduced CoLoC (Conditioned Localizer and Classifier). This two-stage framework decomposes the SELD task into sequential localization and classification steps, effectively managing overlapping sounds without the overhead of PIT.

The CoLoC Framework

CoLoC employs a two-stage pipeline based on Convolutional Recurrent Neural Networks (CRNNs):

  1. Self-Conditioned SSG Localizer: Utilizing Sequential Set Generation (SSG), this module recursively identifies Directions of Arrival (DOAs). It predicts one location and uses it as a "hint" to find the subsequent location, continuing until a "stop" token (the origin) is reached. This recursive nature allows the model to dynamically handle an arbitrary number of sound sources.
  2. Location-Conditioned Classifier: This stage takes the audio features alongside a specific DOA provided by the localizer to predict the class probability for that specific spatial point.

By making the localizer class-agnostic, the system focuses on the physical properties of the sound field before assigning semantic labels, which enhances its ability to handle complex overlaps.

Data Representation: Stacked-Tracks

A key innovation in training CoLoC is the "Stacked-Tracks" meta-data format. In this scheme, active sound events are organized into consistent rows, "stacking" from the bottom up. When an event concludes, higher-level events "fall" to fill the vacancy. This ensures a stable and predictable input structure for the recursive localizer.

The 'Stacked-Tracks' format organizes meta-data into consistent rows to facilitate recursive localization training.

Training Strategies and Robustness

To optimize performance and mitigate the risks of a two-stage architecture, the authors implemented several specific strategies:

  • $L_{1.5}$ Loss for Localization: The authors adopted an $L_{1.5}$ norm for DOA estimation. They found that while $L_2$ loss tends to average multiple sources and $L_1$ loss can be too unstable, $L_{1.5}$ provides an ideal balance for stable training.
  • Angle Perturbation: To prevent error propagation—where small inaccuracies in localization lead to classification failures—the classifier is trained with "Angle Perturbation." By randomly jittering input DOAs by 5 degrees, the classifier becomes more resilient to imperfect localizer outputs.

Experimental Results

The model was evaluated using the STARSS22 dataset of real-world recordings, supplemented by a synthetic dataset of 1,200 mixtures to ensure data density. Results indicate that CoLoC significantly outperforms the DCASE2022 baseline.

Performance comparison of CoLoC variants against the DCASE2022 baseline.

As shown in the results table, the max_ov2 configuration improved the F1-score from 21% to 33% and reduced the Localization Error ($LE_{CD}$) from 29.3° to 24.6°. While the system showed a slightly higher overall Error Rate ($ER_{20^\circ}$) due to the propagation of "phantom" source predictions, its gains in precision and localization accuracy are substantial.

Conclusion

CoLoC demonstrates that decomposing SELD into conditioned stages is a viable and efficient alternative to permutation-invariant approaches. By focusing on class-agnostic localization followed by conditioned classification, the model successfully navigates the complexities of homogeneous sound overlaps. Future research may explore hybridizing Sequential Set Generation with PIT to further reduce error propagation and eliminate false positive "phantom" sources.

Summary updated

Abstract

In this article, we describe Conditioned Localizer and Classifier (CoLoC) which is a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes of two stages: the localization is done first and is followed by classification conditioned by the output of the localizer. In order to resolve the problem of the unknown number of sources we incorporate the idea borrowed from Sequential Set Generation (SSG). Models from both stages are SELDnet-like CRNNs, but with single outputs. Conducted reasoning shows that such two single-output models are fit for SELD task. We show that our solution improves on the baseline system in most metrics on the STARSS22 Dataset.
BibTeX
@inproceedings{kapka2022coloc,
  author = {Slawomir Kapka and Jakub Tkaczuk},
  title = {{CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=H5AWGZWACAW},
}
Figure