Abstract
Jointly learning from a small labeled set and a larger unlabeled set is an active research topic under semi-supervised learning (SSL). In this paper, we propose a novel SSL method based on a two-stage framework for leveraging a large unlabeled in-domain set. Stage-1 of our proposed framework focuses on audio-tagging (AT), which assists the sound event detection (SED) system in Stage-2. The AT system is trained utilizing a strongly labeled set converted into weak predictions referred to as weakified set, a weakly labeled set, and an unlabeled set. This AT system then infers on the unlabeled set to generate reliable pseudo-weak labels, which are used with the strongly and weakly labeled set to train a frequency dynamic convolutional recurrent neural network-based SED system at Stage-2 in a supervised manner. Our system outperforms the baseline by 45.5% in terms of polyphonic sound detection score on the DESED real validation set.
Introduction
Sound Event Detection (SED) involves identifying the presence of specific sounds (e.g., a dog barking) and their precise temporal boundaries (onset and offset). While unlabeled audio data is abundant, obtaining "strong labels" with precise timestamps is labor-intensive and expensive. Conversely, "weak labels"—which indicate the presence of a sound within a clip without timing information—are easier to acquire but provide less granular information.
Existing semi-supervised learning (SSL) methods, such as Mean-Teacher models, often struggle to fully exploit large unlabeled datasets. This research addresses the gap by introducing a two-stage framework that leverages a high-quality audio tagging (AT) stage to generate reliable pseudo-labels, effectively "priming" the subsequent detection stage.
Methodology
The proposed framework operates in two distinct stages to maximize the utility of both labeled and unlabeled data.
Stage-1: Audio Tagging (AT)
Instead of attempting detection immediately, the system first learns to tag the presence of sounds globally.
- Weakified Labels: A core innovation of this work is "weakification." The authors take existing strongly labeled data and strip away the timestamps, treating them as weak labels. This prevents the model from developing a bias toward predicting silence or inactive frames during the initial tagging phase.
- PANNs: The system utilizes Pre-trained Audio Neural Networks (CNN-14 architecture) as a robust feature extractor.
- Pseudo-labeling: The Stage-1 model performs inference on a massive unlabeled dataset to generate "pseudo-weak" labels, which serve as ground truth for the next stage.
Stage-2: Sound Event Detection (SED)
The system is trained for precise temporal localization using the original labeled data combined with the pseudo-weak labels generated in Stage-1.
- Frequency Dynamic (FDY) Convolutions: Standard convolutions assume frequency shift-invariance, treating spectrograms like standard images. However, audio features are often frequency-dependent. FDY-convolutions apply frequency-adaptive kernels to better capture these specific patterns.
- Advanced Loss Functions: To handle the imbalance between active and inactive sound frames, the researchers utilized Asymmetric Focal Loss (AFL) and replaced standard attention pooling with Exponential Softmax.
Experimental Setup and Datasets
The study utilized the DCASE 2022 Task 4 dataset, which simulates domestic environments:
- 1,578 real recordings with weak annotations.
- 14,412 unlabeled in-domain recordings.
- 10,000 synthetic recordings with strong annotations.
- 3,470 real recordings from AudioSet with strong annotations.
Evaluation was conducted on the DESED real validation set and a public YouTube evaluation set using the Polyphonic Sound Detection Score (PSDS).
Key Results and Discussion
The proposed system demonstrated a 45.5% improvement in PSDS over the DCASE 2022 baseline. The results highlight the importance of separating the classification (tagging) and localization (detection) tasks.
Insights
- The Weakification Paradox: Counterintuitively, discarding precise timestamp data to create weak labels for Stage-1 helps the model focus on global sound characteristics, leading to more reliable pseudo-labels for the unlabeled set.
- Frequency Sensitivity: By using FDY-convolutions, the model acknowledges that a sound's identity is tied to its frequency band, allowing for specialized filters across the spectrum.
- Reliability vs. Precision: High performance in Stage-1 (measured by PSDS2, which emphasizes classification) directly correlates to superior final SED performance in Stage-2.
Conclusion and Future Work
This two-stage approach provides a powerful method for leveraging unlabeled data in SED tasks. While the system achieved state-of-the-art results, the authors note that the CNN-14 PANNs architecture is computationally heavy (118 million parameters), suggesting that future work should explore model distillation for edge device deployment. Additionally, further research into hyperparameter sensitivity for Asymmetric Focal Loss and alternative pooling functions could yield even greater performance gains.
Summary updated