Event-Independent Network for Polyphonic Sound Event Localization and Detection

Yin Cao, Turab Iqbal, Qiuqiang Kong, Yue Zhong, Wenwu Wang, Mark D. Plumbley

Introduction

Sound Event Localization and Detection (SELD) is a critical task in machine hearing that requires a system to simultaneously identify the type of a sound (e.g., a dog barking) and its spatial origin (Direction-of-Arrival, or DoA). While significant progress has been made, traditional class-based SELD systems encounter a fundamental limitation: they struggle to distinguish between multiple overlapping instances of the same sound class, such as two people speaking simultaneously from different locations. This is often referred to as the "double talker" problem.

To address this, researchers from the University of Surrey and ByteDance have developed a novel Event-Independent Network. This architecture shifts the paradigm from class-based prediction to a track-based approach, allowing the system to handle complex polyphonic scenarios where sound classes overlap.

Methodology

The proposed system moves away from predicting specific classes directly and instead adopts an end-to-end, track-based prediction framework. The architecture is defined by three primary components:

  1. Feature Extraction: The network processes First-Order Ambisonics (FOA) signals. It utilizes a 1-D convolutional layer to extract log-mel spectrograms for identity and intensity vectors for spatial localization directly within the model pipeline.
  2. Track-Wise Prediction and tPIT: Instead of assigning outputs to fixed classes, the network predicts activity on generic "tracks." To solve the track permutation problem—where a sound might jump between tracks across different frames—the authors implement frame-level Permutation Invariant Training (tPIT). This strategy calculates loss for all possible track assignments and optimizes for the most consistent one, forcing the network to maintain stable track identities.
  3. Event Activity Detection (EAD) Branch: A dedicated EAD branch integrates features from both the Sound Event Detection (SED) and DoA branches. This branch acts as a gatekeeper, improving the precision of event onset and offset detection by leveraging both spectral identity and spatial consistency.
The proposed Event-Independent Network architecture featuring SED, EAD, and DoA branches with tPIT-based loss alignment.

Experimental Results

The model was rigorously evaluated using the TAU-NIGENS Spatial Sound Events 2020 dataset, which includes 700 samples across 14 classes with realistic reverberation and moving sources. The "Event-Ind" model was compared against several DCASE 2020 baselines and track-wise ablation studies.

As shown in the performance metrics, the Event-Independent approach significantly outperformed the baselines. It achieved the lowest Error Rate ($ER_{20^\circ}$) and highest F-score ($F_{20^\circ}$), while also showing superior localization accuracy ($LE_{CD}$) and recall ($LR_{CD}$). The results confirm that the combination of tPIT and the EAD branch is essential for accurately resolving polyphonic sound events.

Performance comparison of the proposed Event-Ind model against DCASE baselines across standard SELD metrics.

Conclusion

The Event-Independent Network provides a robust solution to the challenges of polyphonic SELD. By decoupling detection from specific classes and utilizing track-based prediction with tPIT, the system can effectively localize and identify overlapping sounds of the same type. Future work will look toward extending the system to handle more than two simultaneous tracks and optimizing the architecture for real-time, low-latency applications on edge devices.

Summary updated

Abstract

Polyphonic sound event localization and detection is not only detecting what sound events are happening but localizing corresponding sound sources. This series of tasks was first introduced in DCASE 2019 Task 3. In 2020, the sound event localization and detection task introduces additional challenges in moving sound sources and overlapping-event cases, which include two events of the same type with two different direction-of-arrival (DoA) angles. In this paper, a novel event-independent network for polyphonic sound event localization and detection is proposed. Unlike the two-stage method we proposed in DCASE 2019 Task 3, this new network is fully end-to-end. Inputs to the network are first-order Ambisonics (FOA) time-domain signals, which are then fed into a 1-D convolutional layer to extract acoustic features. The network is then split into two parallel branches. The first branch is for sound event detection (SED), and the second branch is for DoA estimation. There are three types of predictions from the network, SED predictions, DoA predictions, and event activity detection (EAD) predictions that are used to combine the SED and DoA features for on-set and off-set estimation. All of these predictions have the format of two tracks indicating that there are at most two overlapping events. Within each track, there could be at most one event happening. This architecture introduces a problem of track permutation. To address this problem, a frame-level permutation invariant training method is used. Experimental results show that the proposed method can detect polyphonic sound events and their corresponding DoAs. Its performance on the Task 3 dataset is greatly increased as compared with that of the baseline method.
BibTeX
@inproceedings{cao2020event,
  author = {Yin Cao and Turab Iqbal and Qiuqiang Kong and Yue Zhong and Wenwu Wang and Mark D. Plumbley},
  title = {{Event-Independent Network for Polyphonic Sound Event Localization and Detection}},
  booktitle = {DCASE},
  year = {2020},
  url = {https://openreview.net/forum?id=8ynnwF97cz},
}
Figure