Introduction
Sound Event Detection (SED) involves identifying the presence and temporal boundaries of specific sounds within an acoustic environment. This task is inherently complex, requiring a model to distinguish between short-duration foreground events (e.g., a dog bark) and long-duration background contexts (e.g., street noise). While Convolutional Neural Networks (CNNs) are effective at extracting local features and Transformers excel at modeling global dependencies, integrating both capabilities efficiently remains a challenge.
Furthermore, the scarcity of "strongly labeled" data—audio with precise onset and offset timestamps—necessitates the use of semi-supervised learning techniques to leverage "weakly labeled" and unlabeled datasets. This paper presents a Conformer-based architecture that addresses these challenges, achieving state-of-the-art results in the DCASE2020 Task 4 challenge.
Methodology
The core of the proposed system is the replacement of traditional Recurrent Neural Network (RNN) or pure Transformer layers with Conformer blocks. The Conformer architecture integrates multi-head self-attention with feed-forward and convolution modules, enabling the simultaneous modeling of local and global acoustic dependencies.
To maximize performance with limited strong labels, the authors implemented several key strategies:
- Special Tagging Token: Inspired by the
[CLS]token in BERT, a special token is prepended to the feature sequence. Its output is dedicated to weak label prediction (audio tags), while the remaining sequence handles strong label prediction (timestamps). - Semi-Supervised Learning: A "mean teacher" framework is utilized, where a student model learns from unlabeled data by maintaining consistency with a teacher model (an exponential moving average of the student's weights).
- Data Augmentation: The system employs time-shifting to mitigate temporal bias and mixup to smooth decision boundaries by blending audio samples and their corresponding labels.
- Event-Dependent Post-Processing: Thresholds and median filter sizes are optimized individually for each sound class to account for varying temporal structures.
The model was trained on the DCASE2020 Task 4 dataset, which includes 2,584 strongly labeled synthetic clips, 1,578 weakly labeled real clips, and 14,412 unlabeled real clips.
Results and Discussion
The Conformer-based approach demonstrated a significant performance leap over standard architectures. On the validation set, the model achieved an event-based macro F1 score of 50.6%, substantially outperforming the DCASE baseline of 34.8%.
Key Insights
A critical finding was the importance of the receptive field within the Conformer block. Tuning the depth-wise convolution kernel size to 7 (corresponding to a 1.12-second receptive field) proved optimal for capturing local acoustic events.
Visual analysis of the attention weights provided further validation of the architecture's strengths. Unlike pure Transformers, which typically exhibit vertical patterns (global context), the Conformer's attention heads displayed both diagonal patterns (tracking local temporal progression) and vertical patterns.
Furthermore, the study highlighted the synergy between different data augmentation techniques. While Gaussian noise and frequency masking showed limited utility, the combination of time-shifting and mixup yielded statistically significant improvements in robustness.
Challenge Performance
By ensembling eight Conformer models using posterior-level score fusion, the system achieved a Polyphonic Sound Detection Score (PSDS) of 0.700 on the validation set. In the official DCASE2020 challenge, the system secured 1st place with an F1 score of 51.1%.
Conclusion
The integration of Conformer blocks into the SED pipeline effectively bridges the gap between local and global feature extraction. While the system achieves state-of-the-art results, performance varies across sound classes, suggesting that future work should focus on event-wise optimization and the integration of audio source separation to handle overlapping sounds in complex environments.
Summary updated