TL;DR
This paper introduces a new challenge and dataset for general-purpose audio tagging, aiming to classify audio clips into 41 diverse categories from the AudioSet Ontology. They present a baseline system and discuss the unique challenges of working with user-generated audio from Freesound, which often has noisy or incomplete metadata.
What Problem Does This Paper Address?
The paper tackles the challenge of building robust audio tagging systems that can recognize a wide variety of sounds, moving beyond domain-specific applications. While previous DCASE challenges focused on specific sound environments (like domestic or in-car), this work addresses the need for general-purpose audio recognition. A key problem is the availability of large-scale, diverse audio data with reliable annotations. User-generated audio platforms like Freesound offer a wealth of content, but their metadata can be inconsistent, incomplete, or noisy, making it difficult to create high-quality training datasets. This paper aims to bridge this gap by creating a dataset and a challenge that specifically deals with these issues.
Methods
The core of the paper describes the creation of the "Freesound Dataset Kaggle 2018" (FSDKaggle2018) and a baseline system for the DCASE 2018 Task 2.
- Dataset Curation: The dataset was built from Freesound, a large repository of user-uploaded sounds. The process involved:
- Mapping Freesound tags to AudioSet Ontology categories.
- Generating candidate annotations based on this mapping.
- A manual validation step using the Freesound Datasets platform, where users rated the presence of categories in audio clips. This resulted in "manually-verified" annotations.
- A "non-verified" annotation subset was also created, primarily from unrated candidates and a small amount of rated ones, with a quality estimate (QE) of at least 65%.
- Rigorous filtering was applied to remove categories not in leaf nodes of the AudioSet hierarchy, clips with unsuitable licenses or durations, and categories with insufficient data.
- A final dataset of 11,073 audio clips across 41 AudioSet categories was curated, with a focus on single-label classification.
- Baseline System: A relatively shallow 3-layer Convolutional Neural Network (CNN) was used.
- Input: Log-mel spectrograms derived from audio clips.
- Processing: Audio is segmented into overlapping windows. Spectrograms are computed and then log-transformed.
- Architecture: Three 2D convolutional layers with alternating max-pooling layers, followed by a softmax classifier.
- Prediction: Clip-level predictions are obtained by averaging window-level predictions.
Datasets
- Freesound Dataset Kaggle 2018 (FSDKaggle2018): This is the primary dataset introduced for the DCASE 2018 Task 2. It's a subset of the larger Freesound Dataset (FSD).
- Size: 11,073 audio clips.
- Categories: 41 diverse categories from the AudioSet Ontology.
- Annotation Types: * Manually-verified annotations: High-confidence labels.
- Non-verified annotations: Lower confidence, but still useful, with a quality estimate (QE) of at least 65%.
- Clip Characteristics: Variable length (300ms to 30s), uncompressed PCM 16-bit, 44.1 kHz, mono.
- Split: Train set (9473 clips) and test set (1600 clips for evaluation, plus 7800 padding clips).
- AudioSet Ontology: Used as the source for the 41 target categories.
Key Insights and Ideas
- The Challenge of User-Generated Audio: The paper highlights that user-generated audio, while abundant, presents significant challenges due to inconsistent metadata, varying recording quality, and potential label noise. This necessitates new approaches to dataset curation and model training.
- Leveraging Noisy Labels: The introduction of "non-verified" annotations with a known quality estimate is a key idea. This allows participants to potentially leverage more data, even if it's not perfectly clean, by developing methods that can handle varying levels of annotation reliability.
- General-Purpose Tagging is Crucial: The move towards a broader set of sound categories reflects a growing need for audio systems that can understand sounds from diverse real-world environments, not just specialized domains.
- Baseline Performance: The baseline CNN achieved a respectable mAP@3 of 0.70, demonstrating the feasibility of the task and providing a benchmark for future research.
Key Results
- Dataset Creation: A dataset of 11,073 audio clips across 41 categories was successfully curated for the DCASE 2018 Task 2.
- Baseline Performance: The 3-layer CNN baseline achieved a mean Average Precision @ 3 (mAP@3) of 0.70 on the entire test set. This performance was consistent across the public and private Kaggle leaderboard splits (0.70 and 0.69, respectively).
- Category Performance: Per-category AP@3 scores varied significantly, with some categories like "Applause" and "Laughter" achieving high scores (0.98 and 0.96 respectively), while others like "Squeak" and "Drawer open, close" had very low scores (0.16 and 0.05 respectively).
Limitations and Future Work
- Annotation Reliability: The dataset contains a significant portion of "non-verified" annotations, which inherently have some level of noise. The paper acknowledges that these might include out-of-domain sounds.
- Single-Label Assumption: The dataset was curated to focus on single-label classification for simplicity. However, many real-world audio clips contain multiple simultaneous sound events.
- Future Work: The authors suggest that future work could explore methods to better handle the varying reliability of annotations and to extend the system to multi-label audio tagging. They also mention that future updates to the dataset will include ground-truth data for the test set and additional Freesound metadata. The competition itself is intended to foster open research in sound event recognition.
Summary updated