Abstract
Polyphonic Sound Event Detection (SED) often struggles with temporal precision due to overlapping sound sources and the inherent limitations of frame-level classification. This paper introduces a multi-task learning framework that incorporates Sound Activity Detection (SAD) and Onset Detection as auxiliary tasks to guide the primary SED model. By leveraging these temporal anchors through a conditioning hierarchy, the proposed approach achieves a three-fold improvement in event-based F-score on the URBAN-SED dataset compared to standard baseline models.
Introduction
Detecting multiple overlapping sounds in real-world environments—such as car horns, sirens, and jackhammers on a busy street—is the core challenge of polyphonic Sound Event Detection (SED). Current state-of-the-art systems typically treat this as a frame-by-frame classification problem. However, this approach faces two significant hurdles:
- Masking: Overlapping sounds interfere with each other's frequency profiles, making it difficult for classifiers to distinguish individual events.
- Temporal Blur: While frame-level classifiers can identify the presence of a sound within a segment, they often fail to pinpoint the exact start and end times.
In critical applications like audio surveillance or healthcare monitoring, precise onset detection is vital. While segment-based accuracy may appear high, event-based accuracy—which requires precise timing—is often significantly lower, sometimes falling below 10% in baseline systems.
Methodology
The authors propose a Multi-Task Learning (MTL) framework using a Convolutional Recurrent Neural Network (CRNN) architecture. The problem is decomposed into three interdependent tasks:
- SED (The Main Task): Identifying specific labels (e.g., "dog bark") and their durations.
- Sound Activity Detection (SAD): A binary task determining the presence of any sound versus background noise.
- Onset Detection: Identifying the precise beginning of a sound event.
The core innovation is a conditioning hierarchy. Rather than training tasks in isolation, the model uses predictions from Onset and SAD tasks to refine the SED output. For instance, in the sed_sad_onset configuration, the network first predicts onsets, uses those to sharpen its understanding of sound activity, and finally integrates both to produce high-precision sound event predictions.
Experimental Setup
The study utilizes the URBAN-SED dataset, which contains 10,000 soundscapes generated via the Scaper library. Each 10-second clip includes one to nine overlapping urban sounds (e.g., air conditioners, drilling, children playing) against a background of Brownian noise to simulate realistic acoustic environments. The data is split into 6,000 training, 2,000 validation, and 2,000 testing clips.
Results and Discussion
The multi-task approach yields substantial improvements, particularly in metrics requiring temporal precision:
- SED Performance: The baseline SED model achieved an event-based F-score of only 7.34%. By adding onset conditioning (
sed_onset), this rose to 21.42%. The full joint model (sed_sad_onset) reached 23.20%, representing a three-fold relative improvement over the baseline. - SAD Robustness: Unlike specific event detection, SAD is less affected by masking because it only tracks general activity. Conditioning SAD on onsets improved its event-based F-score from 43.14% to 70.31%.
- Temporal Anchoring: Onsets, characterized by sudden energy bursts, serve as reliable temporal anchors. The authors found that conditioning SED on onsets is more effective than conditioning on general activity, as the precise "start" signal is more valuable for event-based evaluation.
While event-based scores improved dramatically, segment-based (coarse) scores occasionally dipped slightly. This indicates the model is becoming more selective and precise, avoiding the "blurry" over-estimations common in frame-level classifiers.
Conclusion
This research demonstrates that multi-task learning with onset and activity conditioning significantly enhances the temporal accuracy of polyphonic SED. Future work will explore task-dependent loss weighting schemes and methods to reduce the model's sensitivity to onset detection errors, which can currently lead to missed events or "ghost" detections.
Summary updated