Abstract
In this paper, we describe in detail our system for DCASE 2022 Task 4. The system combines two considerably different models: an end-to-end Sound Event Detection Transformer (SEDT) and a frame-wise model, Metric Learning and Focal Loss CNN (MLFL-CNN). The former is an event-wise model which learns event-level representations and predicts sound event categories and boundaries directly, while the latter is based on the widely adopted frame-classification scheme. Experimental results show that the hybrid system considerably outperforms either individual model and achieves psds1 of 0.420 and psds2 of 0.783 on the validation set without external data.
Introduction
Sound Event Detection (SED) involves identifying both the category and the precise temporal boundaries (onset and offset) of sound events in audio streams. This task is particularly challenging in domestic environments characterized by overlapping sounds and noisy backgrounds. The DCASE 2022 Task 4 challenge further complicates this by requiring models to learn from heterogeneous data sources, including weakly labeled real data, unlabeled real data, and strongly labeled synthetic data.
Traditional SED approaches typically rely on Convolutional Recurrent Neural Networks (CRNNs) that process audio frame-by-frame. While effective, these models often struggle to capture long-range global context and require extensive post-processing to consolidate frame-level predictions into discrete events. To address these limitations, we propose a hybrid system that leverages the complementary strengths of an event-wise Transformer and a frame-wise CNN.
Methodology
Sound Event Detection Transformer (SEDT)
The SEDT treats sound event detection as a set prediction problem, directly outputting event categories and time boundaries. The architecture consists of a CNN backbone for feature extraction, a Transformer encoder for global context modeling, and a Transformer decoder that utilizes audio and event queries to generate predictions.
To effectively utilize unlabeled data, we employ a semi-supervised Teacher-Student framework. An online "Teacher" model, updated via Exponential Moving Average (EMA) from the "Student," generates reliable pseudo-labels on weakly-augmented audio. The Student model is then trained to match these labels using strongly-augmented versions of the same audio, promoting feature invariance and robustness.
Metric Learning and Focal Loss CNN (MLFL-CNN)
The frame-wise component of our system is based on the MLFL-CNN architecture. This model employs metric learning to bridge the domain gap between synthetic and real-world audio, ensuring that the learned features are consistent across different data distributions.
A key feature of this model is tag-conditioning, where a multi-label classifier predicts global audio tags that are then used to refine the frame-level event detection. This hierarchical approach helps in filtering out false positives that are inconsistent with the overall audio context.
Hybrid Fusion Strategy
The final output is generated by fusing the predictions of the SEDT and MLFL-CNN. Rather than simple averaging, we use a class-specific weighting strategy based on Polyphonic Sound Detection Score (PSDS) performance. This allows the system to favor the model that performs better for specific sound categories (e.g., using SEDT for "Dishes" and MLFL-CNN for "Alarm clocks").
Results and Discussion
The hybrid system was evaluated on the DCASE 2022 Task 4 development set. The combination of event-wise and frame-wise modeling proved highly effective:
- Internal Data Only: The system achieved a PSDS1 of 0.420 and a PSDS2 of 0.783.
- With External Data: Incorporating AudioSet and SINS datasets improved performance to PSDS1 0.449 and PSDS2 0.816.
Our analysis indicates that the SEDT excels at PSDS1, which emphasizes temporal precision, while the MLFL-CNN is superior for PSDS2, which prioritizes classification robustness. The use of asymmetric augmentation and class-wise median filter tuning were also identified as critical factors in achieving these results.
Conclusion
We presented a hybrid SED system that successfully integrates the global modeling capabilities of Transformers with the local precision and domain adaptation of metric-learning CNNs. By combining these two distinct philosophies, our system achieved competitive rankings in the DCASE 2022 Task 4 challenge. Future work will focus on developing unified architectures that can natively perform both event-wise and frame-wise detection to reduce computational overhead.
Summary updated