Online Active Learning For Sound Event Detection

Mark Lindsey, Ankit Shah, Francis Kubala, Richard M. SternarXiv 2023 · DOI · arXiv

Introduction

Training machine learning models for Sound Event Detection (SED)—such as identifying human speech in urban environments or distinguishing live voices in broadcasts—presents a significant challenge. The primary bottleneck is human annotation; labeling audio is cognitively demanding and time-consuming.

Researchers from Carnegie Mellon University and Probity, Inc. have proposed a framework that combines Online Learning with Active Learning (OAL). This approach allows models to begin training almost immediately as data arrives, significantly reducing the volume of labeled data required to achieve high performance.

The Challenge of Sound Event Detection

In traditional supervised learning, practitioners must collect a massive dataset and label it entirely before training begins. While Active Learning (AL) improves efficiency by querying labels for only the most informative data points, it typically assumes a static, pre-collected data pool. This "offline" nature is slow and doesn't account for real-time needs.

Online Active Learning (OAL) addresses two critical real-world issues:

  1. The Cold Start: The need to deploy and train models while data collection is ongoing, rather than waiting for thousands of samples to be annotated.
  2. Data Drift: Real-world audio environments change over time (e.g., urban noise patterns), requiring models to adapt continuously to new acoustic conditions.

Furthermore, standard loss functions like cross-entropy treat all errors equally. In detection tasks, a "miss" (False Negative) is often more costly than a "false alarm" (False Positive). The authors address this by directly optimizing for detection costs.

Methods and Framework

The OAL Workflow

The proposed system processes audio in sequential "sessions" (e.g., 5-minute blocks). The workflow involves:

  • Bootstrapping: Initializing with a minimal dataset to establish a baseline.
  • Query Selection: Using uncertainty sampling (negative energy) to select the $B$ most informative samples per session for human labeling.
  • Continuous Updates: Parameters are updated immediately with new labels before moving to the next session, allowing the model to evolve in real-time.
Comparison of standard Active Learning (top) and the proposed Online Active Learning framework (bottom).

Model Architecture

The system utilizes a Contrastive Siamese Network followed by a Classifier Network. It takes audio embeddings from pre-trained models (e.g., Wav2CLIP or CLAP) and learns a high-dimensional mapping that clusters similar sounds while separating dissimilar ones. This embedding space is then used by the classifier to make final predictions.

The model architecture featuring a Contrastive Siamese Network for embedding and a Classifier Network for prediction.

Differentiable Loss Functions

To optimize for the Detection Cost Function (DCF)—a weighted average of False Negatives and False Positives—the authors introduced two differentiable approximations:

  • e-DCF (Expected DCF): Replaces hard classifications with probabilities to estimate error rates.
  • d-DCF (Differentiable DCF): Employs a "differentiable argmax" (Softmax with a high scaling factor) to approximate the true DCF more closely.

Evaluation and Results

The framework was evaluated on the SONYC Urban Sound Tagging dataset (human speech detection in NYC) and the VTD (Voice-Type Discrimination) corpora (live vs. recorded speech).

Key Findings

  • Efficiency: On the SONYC dataset, OAL achieved performance comparable to full supervision using 5x fewer labeled samples. For VTD, the model reached a low DCF (0.0718) using only 1.1% of available labels.
  • Early Training: Models could begin training with as few as 30 samples, compared to the thousands required for traditional offline methods.
  • Loss Function Performance: The d-DCF and e-DCF losses reduced the False Negative Rate (FNR) by up to 30% in supervised and batch-active learning settings by allowing practitioners to weight misses more heavily than false alarms.

Conclusion and Future Directions

The OAL framework demonstrates that sound detectors can be trained efficiently with minimal human intervention. While the DCF-based loss functions showed significant gains in batch settings, their performance in the high-variance online setting remains an area for further research. Future work will focus on refining query strategies for extreme data drift and scaling to multi-class detection scenarios.

Summary updated

BibTeX
@misc{lindsey2023online,
  author = {Mark Lindsey and Ankit Shah and Francis Kubala and Richard M. Stern},
  title = {{Online Active Learning For Sound Event Detection}},
  year = {2023},
  eprint = {2309.14460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.14460},
}
Figure