Abstract
Polyphonic sound event localization and detection is not only detecting what sound events are happening but localizing corresponding sound sources. This series of tasks was first introduced in DCASE 2019 Task 3. In 2020, the sound event localization and detection task introduces additional challenges in moving sound sources and overlapping-event cases, which include two events of the same type with two different direction-of-arrival (DoA) angles. In this paper, a novel event-independent network for polyphonic sound event localization and detection is proposed. Unlike the two-stage method we proposed in DCASE 2019 Task 3, this new network is fully end-to-end. Inputs to the network are first-order Ambisonics (FOA) time-domain signals, which are then fed into a 1-D convolutional layer to extract acoustic features. The network is then split into two parallel branches. The first branch is for sound event detection (SED), and the second branch is for DoA estimation. There are three types of predictions from the network, SED predictions, DoA predictions, and event activity detection (EAD) predictions that are used to combine the SED and DoA features for on-set and off-set estimation. All of these predictions have the format of two tracks indicating that there are at most two overlapping events. Within each track, there could be at most one event happening. This architecture introduces a problem of track permutation. To address this problem, a frame-level permutation invariant training method is used. Experimental results show that the proposed method can detect polyphonic sound events and their corresponding DoAs. Its performance on the Task 3 dataset is greatly increased as compared with that of the baseline method.
Introduction
Sound Event Localization and Detection (SELD) involves identifying the type of sound events (SED) and their spatial coordinates (DoA). A significant challenge in SELD is handling polyphony, particularly when multiple sound events of the same class overlap (e.g., two simultaneous speakers). Traditional class-based detection systems often fail in these scenarios because they are designed to output a single probability per class, making it impossible to distinguish between two distinct instances of the same category.
Methodology
The proposed Event-Independent Network (EIN) addresses the polyphony problem by shifting from class-based to track-wise prediction. The architecture consists of three main branches:
- SED Branch: Predicts the sound event category for each track.
- DoA Branch: Estimates the azimuth and elevation angles for the sound sources.
- EAD (Event Activity Detection) Branch: A novel addition that combines features from both SED and DoA branches to predict the presence of an event. This branch acts as a mask to refine on-set and off-set estimation.
To handle the ambiguity of which sound event is assigned to which track, the model employs frame-level Permutation Invariant Training (tPIT). During training, the loss is calculated for all possible track-to-label permutations, and the network is optimized based on the minimum loss. This ensures consistent track assignment across frames, even when events overlap or cross paths.
Results and Discussion
The EIN model was evaluated on the TAU-NIGENS Spatial Sound Events 2020 dataset. The results demonstrate a substantial improvement over baseline methods across all primary metrics.
As shown in the figure above, the "Event-Ind" model achieves the lowest Error Rate ($ER_{20^\circ} = 0.47$) and the highest F-score ($F_{20^\circ} = 61.5\%$). Furthermore, it significantly reduces the Localization Error ($LE_{CD}$) to 16.7° while maintaining a high Localization Recall ($LR_{CD}$) of 75.4%. Ablation studies highlighted that the synergy between the EAD branch and tPIT is critical for these performance gains, particularly in resolving track permutation issues in complex acoustic environments.
Conclusion
The Event-Independent Network represents a significant step forward in polyphonic SELD. By decoupling predictions from specific sound classes and utilizing track-wise outputs with permutation-invariant training, the system effectively handles overlapping events of the same type. Future work will explore expanding the number of tracks and improving the computational efficiency of the tPIT process for real-time applications.
Summary updated