A Hybrid System of Sound Event Detection Transformer and Frame-wise Model for DCASE 2022 Task 4

Yiming Li, Zhifang Guo, Zhirong Ye, Xiangdong Wang, Hong Liu, Yueliang Qian, Rui Tao, Long Yan, Kazushige Ouchi

Abstract

In this paper, we describe in detail our system for DCASE 2022 Task 4. The system combines two considerably different models: an end-to-end Sound Event Detection Transformer (SEDT) and a frame-wise model, Metric Learning and Focal Loss CNN (MLFL-CNN). The former is an event-wise model which learns event-level representations and predicts sound event categories and boundaries directly, while the latter is based on the widely adopted frame-classification scheme. Experimental results show that the hybrid system considerably outperforms either individual model and achieves psds1 of 0.420 and psds2 of 0.783 on the validation set without external data.

Introduction

Sound Event Detection (SED) involves identifying both the category and the precise temporal boundaries (onset and offset) of sound events in audio streams. This task is particularly challenging in domestic environments characterized by overlapping sounds and noisy backgrounds. The DCASE 2022 Task 4 challenge further complicates this by requiring models to learn from heterogeneous data sources, including weakly labeled real data, unlabeled real data, and strongly labeled synthetic data.

Traditional SED approaches typically rely on Convolutional Recurrent Neural Networks (CRNNs) that process audio frame-by-frame. While effective, these models often struggle to capture long-range global context and require extensive post-processing to consolidate frame-level predictions into discrete events. To address these limitations, we propose a hybrid system that leverages the complementary strengths of an event-wise Transformer and a frame-wise CNN.

Methodology

Sound Event Detection Transformer (SEDT)

The SEDT treats sound event detection as a set prediction problem, directly outputting event categories and time boundaries. The architecture consists of a CNN backbone for feature extraction, a Transformer encoder for global context modeling, and a Transformer decoder that utilizes audio and event queries to generate predictions.

Overview of the SEDT architecture, showing the flow from spectrogram to event objects.

To effectively utilize unlabeled data, we employ a semi-supervised Teacher-Student framework. An online "Teacher" model, updated via Exponential Moving Average (EMA) from the "Student," generates reliable pseudo-labels on weakly-augmented audio. The Student model is then trained to match these labels using strongly-augmented versions of the same audio, promoting feature invariance and robustness.

Metric Learning and Focal Loss CNN (MLFL-CNN)

The frame-wise component of our system is based on the MLFL-CNN architecture. This model employs metric learning to bridge the domain gap between synthetic and real-world audio, ensuring that the learned features are consistent across different data distributions.

The MLFL-CNN framework, illustrating the training process with pseudo-labels and tag-conditioned prediction.

A key feature of this model is tag-conditioning, where a multi-label classifier predicts global audio tags that are then used to refine the frame-level event detection. This hierarchical approach helps in filtering out false positives that are inconsistent with the overall audio context.

Hybrid Fusion Strategy

The final output is generated by fusing the predictions of the SEDT and MLFL-CNN. Rather than simple averaging, we use a class-specific weighting strategy based on Polyphonic Sound Detection Score (PSDS) performance. This allows the system to favor the model that performs better for specific sound categories (e.g., using SEDT for "Dishes" and MLFL-CNN for "Alarm clocks").

Results and Discussion

The hybrid system was evaluated on the DCASE 2022 Task 4 development set. The combination of event-wise and frame-wise modeling proved highly effective:

  • Internal Data Only: The system achieved a PSDS1 of 0.420 and a PSDS2 of 0.783.
  • With External Data: Incorporating AudioSet and SINS datasets improved performance to PSDS1 0.449 and PSDS2 0.816.

Our analysis indicates that the SEDT excels at PSDS1, which emphasizes temporal precision, while the MLFL-CNN is superior for PSDS2, which prioritizes classification robustness. The use of asymmetric augmentation and class-wise median filter tuning were also identified as critical factors in achieving these results.

Conclusion

We presented a hybrid SED system that successfully integrates the global modeling capabilities of Transformers with the local precision and domain adaptation of metric-learning CNNs. By combining these two distinct philosophies, our system achieved competitive rankings in the DCASE 2022 Task 4 challenge. Future work will focus on developing unified architectures that can natively perform both event-wise and frame-wise detection to reduce computational overhead.

Summary updated

Abstract

In this paper, we describe in detail our system for DCASE 2022 Task4. The system combines two considerably different models: an end-to-end Sound Event Detection Transformer (SEDT) and a frame-wise model, Metric Learning and Focal Loss CNN (MLFL-CNN). The former is an event-wise model which learns event-level representations and predicts sound event categories and boundaries directly, while the latter is based on the widely adopted frame-classification scheme, under which each frame is classified into event categories and event boundaries are obtained by post-processing such as thresholding and smoothing. For SEDT, self-supervised pre-training using unlabeled data is applied, and semi-supervised learning is adopted by using an online teacher, which is updated from the student model using the Exponential Moving Average (EMA) strategy and generates reliable pseudo labels for weakly-labeled and unlabeled data. For the frame-wise model, the ICT-TOSHIBA system of DCASE 2021 Task 4 is used. Experimental results show that the hybrid system considerably outperforms either individual model and achieves psds1 of 0.420 and psds2 of 0.783 on the validation set without external data. The code is available at https://github.com/965694547/Hybrid-system-of-frame-wise-model-and-SEDT.
BibTeX
@inproceedings{li2022hybrid,
  author = {Yiming Li and Zhifang Guo and Zhirong Ye and Xiangdong Wang and Hong Liu and Yueliang Qian and Rui Tao and Long Yan and Kazushige Ouchi},
  title = {{A Hybrid System of Sound Event Detection Transformer and Frame-wise Model for DCASE 2022 Task 4}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=A4n04l1dzH},
}
Figure