Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data

Qiuqiang Kong, Yong Xu, Iwona Sobieraj, Wenwu Wang, Mark D. Plumbley

Abstract

Sound event detection (SED) aims to detect when and recognize what sound events happen in an audio clip. Many supervised SED algorithms rely on strongly labelled data which contains the onset and offset annotations of sound events. However, many audio tagging datasets are weakly labelled, that is, only the presence of the sound events is known, without knowing their onset and offset annotations. In this paper, we propose a time-frequency (T-F) segmentation framework trained on weakly labelled data to tackle the sound event detection and separation problem. In training, a segmentation mapping is applied on a T-F representation, such as log mel spectrogram of an audio clip to obtain T-F segmentation masks of sound events. The T-F segmentation masks can be used for separating the sound events from the background scenes in the time-frequency domain. Then a classification mapping is applied on the T-F segmentation masks to estimate the presence probabilities of the sound events. We model the segmentation mapping using a convolutional neural network and the classification mapping using a global weighted rank pooling (GWRP). In SED, predicted onset and offset times can be obtained from the T-F segmentation masks. As a byproduct, separated waveforms of sound events can be obtained from the T-F segmentation masks. We remixed the DCASE 2018 Task 1 acoustic scene data with the DCASE 2018 Task 2 sound events data. When mixing under 0 dB, the proposed method achieved F1 scores of 0.534, 0.398 and 0.167 in audio tagging, frame-wise SED and event-wise SED, outperforming the fully connected deep neural network baseline of 0.331, 0.237 and 0.120, respectively. In T-F segmentation, we achieved an F1 score of 0.218, where previous methods were not able to do T-F segmentation.

Introduction

Sound Event Detection (SED) involves identifying both the nature and the timing of acoustic events within a recording. While strong supervision using onset and offset timestamps is effective, the high cost of manual annotation makes weakly labelled data—where only the presence of a sound is tagged—a more scalable alternative. However, localizing sounds from weak labels is challenging, particularly in the time-frequency (T-F) domain. Most existing weakly supervised methods provide only temporal activations, failing to isolate specific frequency components. This paper introduces a framework that learns T-F segmentation masks from weak labels, enabling both precise event detection and high-quality sound separation.

Proposed Framework

The core of the proposed method is a two-stage mapping architecture designed to translate clip-level tags into localized T-F masks.

Segmentation and Classification Mapping

The framework consists of two primary components:

  1. Segmentation Mapping ($g_1$): A Convolutional Neural Network (CNN) processes the log mel spectrogram as input and outputs T-F segmentation masks. These masks act as saliency maps, identifying the specific T-F units belonging to each sound class.
  2. Classification Mapping ($g_2$): This stage pools the T-F masks to generate final class probabilities. The authors propose Global Weighted Rank Pooling (GWRP) to address the limitations of standard pooling. By sorting T-F units and applying descending weights, GWRP avoids the underestimation of duration typical of max pooling and the noise sensitivity of average pooling.
The proposed framework utilizes segmentation mapping ($g_1$) to generate masks and classification mapping ($g_2$) for final predictions.

System Architecture and Applications

The learned T-F masks serve as a versatile representation for multiple downstream tasks. As shown in the system pipeline, the framework supports:

  • Audio Tagging: Direct output of the classification mapping.
  • Sound Event Detection: Obtained by averaging the T-F masks along the frequency axis and applying post-processing to determine onset and offset times.
  • Sound Event Separation: By upsampling the learned masks and applying them to the original complex spectrogram, individual sound sources can be reconstructed via an Inverse Fourier Transform.
Overview of the system pipeline for training, sound event detection, and sound event separation.

Experimental Results

The researchers evaluated the framework using a remixed dataset of DCASE 2018 Task 1 (background) and Task 2 (foreground) sounds. In high-noise environments (0 dB SNR), the GWRP-based model significantly outperformed baseline deep neural networks.

Metric Proposed (GWRP) Baseline (DNN)
Audio Tagging F1 0.534 0.331
Frame-wise SED F1 0.398 0.237
Event-wise SED F1 0.167 0.120

Furthermore, the model achieved a T-F segmentation F1 score of 0.218. Qualitative analysis confirmed that the learned masks effectively capture the harmonic and temporal structures of diverse sound classes, closely resembling ground-truth Ideal Ratio Masks.

Conclusion

This research demonstrates that weak labels are sufficient for learning complex time-frequency structures. The proposed T-F segmentation framework not only improves sound event detection in noisy conditions but also enables "blind" source separation as a byproduct of the classification task. Future work will explore refining the segmentation mapping to improve separation fidelity and handling overlapping sound events.

Summary updated

Abstract

Sound event detection (SED) aims to detect when and recognize what sound events happen in an audio clip. Many supervised SED algorithms rely on strongly labelled data which contains the onset and offset annotations of sound events. However, many audio tagging datasets are weakly labelled, that is, only the presence of the sound events is known, without knowing their onset and offset annotations. In this paper, we propose a time-frequency (T-F) segmentation framework trained on weakly labelled data to tackle the sound event detection and separation problem. In training, a segmentation mapping is applied on a T-F representation, such as log mel spectrogram of an audio clip to obtain T-F segmentation masks of sound events. The T-F segmentation masks can be used for separating the sound events from the background scenes in the time-frequency domain. Then a classification mapping is applied on the T-F segmentation masks to estimate the presence probabilities of the sound events. We model the segmentation mapping using a convolutional neural network and the classification mapping using a global weighted rank pooling (GWRP). In SED, predicted onset and offset times can be obtained from the T-F segmentation masks. As a byproduct, separated waveforms of sound events can be obtained from the T-F segmentation masks. We remixed the DCASE 2018 Task 1 acoustic scene data with the DCASE 2018 Task 2 sound events data. When mixing under 0 dB, the proposed method achieved F1 scores of 0.534, 0.398 and 0.167 in audio tagging, frame-wise SED and event-wise SED, outperforming the fully connected deep neural network baseline of 0.331, 0.237 and 0.120, respectively. In T-F segmentation, we achieved an F1 score of 0.218, where previous methods were not able to do T-F segmentation.
BibTeX
@article{kong2019sound,
  author = {Qiuqiang Kong and Yong Xu and Iwona Sobieraj and Wenwu Wang and Mark D. Plumbley},
  title = {{Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=dEXyKeSwHa},
}
Figure