A Hybrid System of Sound Event Detection Transformer and Frame-Wise Model for DCASE 2022 Task 4

Yiming Li, Zhifang Guo, Zhirong Ye, Xiangdong Wang, Hong Liu, Yueliang Qian, Rui Tao, Long Yan, Kazushige Ouchi

Abstract

This paper presents a high-performance hybrid system for Sound Event Detection (SED) developed for the DCASE 2022 Task 4 challenge. The system integrates an event-wise Sound Event Detection Transformer (SEDT) with a frame-wise CNN model (MLFL-CNN). By leveraging self-supervised pre-training, teacher-student learning with asymmetric augmentation, and a class-specific fusion strategy, the proposed approach effectively utilizes weakly labeled and unlabeled data. Experimental results demonstrate significant performance gains, achieving high Polyphonic Sound Detection Scores (PSDS) across different evaluation scenarios.

Introduction

Sound Event Detection (SED) involves identifying both the category and the precise temporal boundaries (onset and offset) of sound events in audio clips. The DCASE 2022 Task 4 challenge is particularly demanding due to its reliance on weakly labeled and unlabeled real-world data alongside perfectly labeled synthetic data. Traditional frame-wise models often struggle to capture the global temporal structure of sound events. This work addresses these challenges by proposing a hybrid system that fuses event-wise and frame-wise modeling philosophies to capture both local precision and global context.

Methodology

Sound Event Detection Transformer (SEDT)

The SEDT is an event-wise model that treats sound events as distinct objects. It utilizes a CNN backbone to extract features from spectrograms, which are then processed by a Transformer encoder-decoder to directly predict a set of candidate events.

The architecture of the Sound Event Detection Transformer (SEDT), showing the flow from spectrogram to event-wise predictions.

To enhance the SEDT's performance, several novel techniques were implemented:

  • Self-supervised Pre-training (SP-SEDT): The model learns to predict the location of random spectrogram crops, improving feature representation before supervised training.
  • Teacher-Student Learning: An "online teacher" model, updated via Exponential Moving Average (EMA), generates pseudo-labels for unlabeled data. The system employs asymmetric augmentation, where the teacher processes "easy" audio while the student is challenged with "hard" augmented versions.

MLFL-CNN and Fusion Strategy

The second component is the MLFL-CNN, a frame-wise model based on Metric Learning and Focal Loss. Metric learning is used to bridge the domain gap between synthetic and real audio, while focal loss addresses class imbalance issues common in environmental sound datasets.

The final output is generated through a class-specific fusion method. Rather than simple averaging, weights are assigned to each model based on their performance for specific sound categories (e.g., "Speech" vs. "Vacuum cleaner") on the development set. This allows the system to leverage the specific architectural strengths of each model for different sound types.

Experiments and Results

The system was evaluated using the DCASE 2022 Task 4 framework, incorporating external data from AudioSet and the SINS dataset. Performance was measured using the Polyphonic Sound Detection Score (PSDS), where PSDS1 rewards temporal precision and PSDS2 rewards class macro-discovery.

The hybrid system outperformed individual models significantly. System 1 (optimized for precision) achieved a PSDS1 of 0.449 and PSDS2 of 0.645. System 2 (optimized for detection) achieved a massive PSDS2 of 0.816.

Ablation Study

An ablation study on the SEDT model highlights the contribution of various semi-supervised learning techniques to the overall performance.

Ablation study of the SEDT model showing the impact of Mixup (MU), Focal Loss (FL), Asymmetric Augmentation (AA), and EMA on PSDS scores.

As shown in the table, the combination of Mixup, Focal Loss, and Asymmetric Augmentation significantly boosts performance. For instance, the inclusion of Mixup (comparing the first and last rows) raised the PSDS1 from 0.372 to 0.388, validating the effectiveness of the proposed training pipeline.

Conclusion

The proposed hybrid system successfully combines the complementary strengths of event-wise Transformers and frame-wise CNNs. The integration of advanced semi-supervised learning techniques and class-specific fusion allows the model to excel in both temporal precision and event discovery. Future work will focus on improving the computational efficiency of the ensemble and reducing its reliance on synthetic data quality.

Summary updated

Abstract

In this paper, we describe in detail our system for DCASE 2022 Task4. The system combines two considerably different models: an end-to-end Sound Event Detection Transformer (SEDT) and a frame-wise model, Metric Learning and Focal Loss CNN (MLFL-CNN). The former is an event-wise model which learns event-level representations and predicts sound event categories and boundaries directly, while the latter is based on the widely adopted frame-classification scheme, under which each frame is classified into event categories and event boundaries are obtained by post-processing such as thresholding and smoothing. For SEDT, self-supervised pre-training using unlabeled data is applied, and semi-supervised learning is adopted by using an online teacher, which is updated from the student model using the Exponential Moving Average (EMA) strategy and generates reliable pseudo labels for weakly-labeled and unlabeled data. For the frame-wise model, the ICT-TOSHIBA system of DCASE 2021 Task 4 is used. Experimental results show that the hybrid system considerably outperforms either individual model and achieves psds1 of 0.420 and psds2 of 0.783 on the validation set without external data. The code is available at https://github.com/965694547/Hybrid-system-of-frame-wise-model-and-SEDT.
BibTeX
@inproceedings{li2022hybrid,
  author = {Yiming Li and Zhifang Guo and Zhirong Ye and Xiangdong Wang and Hong Liu and Yueliang Qian and Rui Tao and Long Yan and Kazushige Ouchi},
  title = {{A Hybrid System of Sound Event Detection Transformer and Frame-Wise Model for DCASE 2022 Task 4}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=A4n04l1dzH},
}
Figure