DCASE2017 Challenge Setup: Tasks, Datasets and Baseline System

Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, Tuomas Virtanen

Introduction

While computer vision has advanced rapidly, "machine hearing"—the computational analysis of acoustic environments—remains a significant frontier. Real-world audio is characterized by overlapping sound events, environmental distortions, and the scarcity of certain critical events like gunshots or glass breaking. A major bottleneck in the field is the high cost of producing "strong labels" (precise onset and offset timestamps). The DCASE 2017 challenge addresses these issues by providing a standardized framework, unified datasets, and a baseline system to benchmark progress in scene classification and sound event detection, with a particular focus on weakly supervised learning.

Methodology

To ensure a level playing field, the authors provide a baseline system utilizing standard features and a simple neural architecture.

Feature Extraction

All tasks utilize log mel-band energies. Audio signals are analyzed in 40ms frames with a 50% overlap, mapped to 40 mel bands. To incorporate temporal context, the system aggregates a 5-frame window, resulting in a 200-length feature vector for the model.

System Architecture

The baseline model is a Multilayer Perceptron (MLP) consisting of two dense hidden layers with 50 units each. Dropout (20%) is applied to mitigate overfitting. The output layer varies by task:

  • Task 1 (Classification): Employs a Softmax layer for mutually exclusive scene categories.
  • Tasks 2-4 (Detection/Tagging): Utilize Sigmoid units to support multi-label classification, allowing for the detection of overlapping sound events.
Illustration of a Sound Event Detection system identifying multiple overlapping events with precise timestamps.

Challenge Tasks and Datasets

The 2017 challenge is organized into four distinct tasks:

  1. Acoustic Scene Classification (TUT Acoustic Scenes 2017): Classifying 10-second audio segments into 15 categories (e.g., "Beach," "Office").
  2. Rare Sound Event Detection: Detecting infrequent events (baby cry, glass break, gun shot) synthesized into background noise.
  3. Sound Event Detection in Real-life Audio (TUT Sound Events 2017): Identifying six target classes in complex street recordings.
  4. Large-scale Weakly Supervised Sound Event Detection (AudioSet Subset): Training on YouTube-derived data with "weak labels" (video-level tags) rather than precise timestamps.

Results and Discussion

The baseline performance highlights the varying difficulty levels of these tasks. In Task 1, the system achieved an overall accuracy of 74.8% on the development set, which fell to 61.0% on the evaluation set, indicating challenges in generalization.

Baseline performance for Acoustic Scene Classification across different environments, showing a significant drop from development to evaluation sets.

Other tasks proved even more difficult. Rare event detection performance varied by class, with "glass break" being the easiest (0.22 Error Rate) and "gun shots" the hardest (0.69 Error Rate). Real-life sound event detection (Task 3) yielded a high Error Rate of 0.93 on the evaluation set. Most notably, the weakly supervised task (Task 4) resulted in a low 10.9% F-score, underscoring the limitations of simple MLPs when dealing with untagged temporal data.

Conclusion

The DCASE 2017 challenge establishes a rigorous benchmark for the machine hearing community. While the MLP baseline provides a necessary starting point, the results suggest that more advanced architectures, such as Convolutional Recurrent Neural Networks (CRNNs), are required to capture the complex spectral and temporal dependencies of real-world audio. The shift toward weakly supervised learning remains the most significant hurdle for scaling these systems to "in-the-wild" data.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{mesaros2017dcase,
  author = {Annamaria Mesaros and Toni Heittola and Aleksandr Diment and Benjamin Elizalde and Ankit Shah and Emmanuel Vincent and Bhiksha Raj and Tuomas Virtanen},
  title = {{DCASE2017 Challenge Setup: Tasks, Datasets and Baseline System}},
  booktitle = {DCASE},
  year = {2017},
  url = {https://openreview.net/forum?id=Sh9j5d4k96I},
}
Figure