Guided multi-branch learning systems for sound event detection with sound separation

Yuxin Huang, Liwei Lin, Shuo Ma, Xiangdong Wang, Hong Liu, Yueliang Qian, Min Liu, Kazushige Ouch

Abstract

This paper presents a sophisticated system for Sound Event Detection (SED) in domestic environments, developed for the DCASE 2020 Task 4 challenge. Building on a teacher-student "guided learning" framework, the authors introduce Multi-Branch Learning (MBL) to improve feature representation by utilizing diverse pooling strategies. The system effectively integrates weakly labeled, unlabeled, and synthetic data, while also incorporating Sound Separation (SS) to enhance performance in noisy acoustic scenes. Experimental results demonstrate that MBL reduces overfitting and that fusing sound-separated models into an ensemble provides a robust performance boost.

Introduction: Addressing Data Scarcity and Signal Interference

Sound Event Detection (SED) involves identifying the presence and temporal boundaries of specific sounds. This task is inherently difficult due to the high cost of obtaining "strongly labeled" data (precise timestamps). Most available datasets consist of "weakly labeled" clips (tags only) or large volumes of unlabeled audio. Additionally, domestic environments are characterized by overlapping sounds and background noise, which can significantly degrade detection accuracy.

The authors aim to overcome these hurdles by:

  1. Maximizing Data Utility: Effectively combining small amounts of synthetic and weakly labeled data with massive unlabeled datasets.
  2. Mitigating Interference: Using Sound Separation (SS) to "clean" audio signals before detection, thereby improving performance in complex acoustic environments.

Methodology: Guided and Multi-Branch Learning

The system architecture is based on a "guided learning" approach, where a Professional Teacher (PT-model) with a large receptive field generates pseudo-labels for unlabeled data. These labels then guide the training of a Promising Student (PS-model), which is optimized for frame-level detection.

Multi-Branch Learning (MBL)

A core innovation is the Multi-Branch Learning structure within the PS-model. Multiple branches share a single feature encoder but employ different pooling strategies to capture varied signal characteristics:

  • Attention-based Pooling (ATP): Learns to focus on the most relevant parts of the audio signal.
  • Global Max/Average Pooling (GMP/GAP): Provides statistical summaries of features.
  • Sound Event Detection Branch (SEDB): Leverages strongly labeled synthetic data to predict instance-level probabilities.
Figure 1: The multi-branch learning architecture, illustrating how different label types flow through the shared feature encoder and specialized branches.

By training the encoder to satisfy multiple pooling objectives, the model learns a more generalized feature space, which helps prevent overfitting to specific data subsets.

Sound Separation Integration

To further improve robustness, the authors integrated a Sound Separation (SS) module. Audio is first separated into distinct tracks (e.g., isolating speech from background noise), and dedicated SED models are trained on this processed audio. While these SS-SED models are less accurate on their own, they provide a unique perspective that enhances the overall performance when fused into a final ensemble.

Key Results and Insights

The system was evaluated on the DCASE 2020 Task 4 dataset using the event-based macro F1 score.

Figure 6: Performance metrics for various model configurations, highlighting the improvements gained from multi-branch learning and ensemble fusion.

Performance Gains

  • Pooling Synergy: Adding the MBL structure (specifically the I-GAP branch) improved the validation F1 score from 0.444 to 0.451. This confirms that diverse pooling methods capture complementary information.
  • Ensemble Performance: The best ensemble achieved an F1 score of 44.6% for standard SED and 44.7% when integrated with sound separation. The inclusion of sound-separated models helped the system generalize better across different acoustic events.
  • Batch Balancing: The authors found that MBL is sensitive to the ratio of unlabeled data. Carefully balancing labeled and unlabeled data in each mini-batch was crucial to prevent noisy pseudo-labels from degrading performance.

Conclusion and Future Directions

The combination of multi-branch learning and sound separation offers a powerful framework for SED in complex, real-world environments. While current sound separation systems can introduce artifacts that limit the performance of individual SS-SED models, their contribution to ensemble diversity is significant. Future research will focus on joint, end-to-end training of separation and detection modules to further refine these systems.

Summary updated

Abstract

In this paper, we describe in detail our systems for DCASE 2020 Task 4. The systems are based on the 1st-place system of DCASE 2019 Task 4, which adopts weakly-supervised framework with an attention-based embedding-level pooling module and a semi-supervised learning approach named guided learning. This year, we incorporate multi-branch learning (MBL) into the original system to further improve its performance. MBL uses different branches with different pooling strategies (including instance-level and embedding-level strategies) and different pooling modules (including attention pooling, global max pooling or global average pooling modules), which share the same feature encoder of the model. Therefore, multiple branches pursuing different purposes and focusing on different characteristics of the data can help the feature encoder model the feature space better and avoid over-fitting. To better exploit the strongly-labeled synthetic data, inspired by multi-task learning, we also employ a sound event detection branch. To combine sound separation (SS) with sound event detection (SED), we fuse the results of SED systems with SS-SED systems which are trained using separated sound output by an SS system. The experimental results prove that MBL can improve the model performance and using SS has great potential to improve the performance of SED ensemble system.
BibTeX
@misc{huang2020guided,
  author = {Yuxin Huang and Liwei Lin and Shuo Ma and Xiangdong Wang and Hong Liu and Yueliang Qian and Min Liu and Kazushige Ouch},
  title = {{Guided multi-branch learning systems for sound event detection with sound separation}},
  year = {2020},
  eprint = {2007.10638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.10638},
}
Figure