The Benefit of Temporally-Strong Labels in Audio Event Classification

Shawn Hershey, Daniel P. W. Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R. Channing Moore, Manoj PlakalICASSP 2021 · DOI

TL;DR

This paper introduces "temporally-strong" labels for audio event classification, providing precise start and end times for sounds, and demonstrates their significant benefit when combined with existing weakly labeled data. By fine-tuning models on even a small set of these strong labels, performance on audio event detection tasks improves substantially, even when evaluated using the original imprecise labels.

What Problem Does This Paper Address?

Training deep learning models for audio event classification typically relies on large datasets with "weak" labels. These labels indicate the presence or absence of a sound within a relatively long time window (e.g., 10 seconds), but lack precise temporal information. This imprecision can limit classifier performance, as sound events often occupy only a fraction of the labeled segment. While datasets with "strong" labels (precise start and end times) exist, they are usually small due to the high cost of manual annotation. This paper addresses the challenge of how to effectively leverage the limited but precise temporal information from strong labels to improve audio event classification models trained on vast amounts of weakly labeled data.

Methods

The core contribution of this work is the creation and utilization of a new dataset with temporally-strong labels for audio events. The authors developed an annotation interface that allows annotators to precisely mark the start and end times of sound events on a spectrogram. This process also included collecting "explicit negatives," where annotators specifically confirmed the absence of certain sounds, to create a more challenging and informative evaluation set.

For training, they experimented with several strategies:

  • Weak-1.8M: The original, large-scale AudioSet dataset with weak labels.
  • Strong-67k: A newly created dataset with temporally-strong labels.
  • Weak-67k: A subset of the original dataset, matched to the clips used for strong labeling, to control for label interpretation differences.
  • Diffuse-67k: A version of the strong labels expanded to the full 10-second clip duration, to isolate the effect of temporal precision.

They then fine-tuned a ResNet-50 model pre-trained on Weak-1.8M using mixtures of this large dataset with either Diffuse-67k or Strong-67k. The mixing ratio was optimized using a validation set.

Datasets

  • AudioSet: The primary dataset used. The authors leverage both its original weak labels and a newly annotated subset.
  • Newly Introduced Strong Labels: The paper introduces a new set of temporally-strong labels for a portion of AudioSet. This dataset includes both positive event annotations with precise start/end times and explicit negative annotations. It also includes "complementary negatives" for evaluation, which are frames surrounding positive events that do not contain the event.

Key Insights and Ideas

  • The Power of Precision: Even a small amount of temporally precise labeling (Strong-67k) can significantly boost classifier performance when combined with a large weakly labeled dataset. This suggests that temporal information is a critical, yet often underutilized, feature in audio event classification.
  • Bridging the Gap: Fine-tuning a model pre-trained on a massive weak dataset with a small strong dataset is a highly effective strategy. This approach leverages the breadth of the weak data and the depth of the strong data.
  • Label Mismatch Matters: Differences in how annotators interpret labels between the original weak labels and the new strong labels can impact evaluation metrics. This highlights the importance of careful dataset curation and potentially automatic label canonicalization.
  • Explicit Negatives Improve Evaluation: Using explicitly marked negative examples, rather than just assuming absence, provides a more realistic and challenging evaluation of classifier performance, particularly for distinguishing confusable sounds.

Key Results

  • Fine-tuning a ResNet-50 model pre-trained on Weak-1.8M with a mixture of Weak-1.8M and Strong-67k achieved the best performance.
  • On the strong evaluation set, this approach improved the d' metric from 1.13 to 1.39.
  • Even when evaluated on the original weak labels, the fine-tuned model showed an improvement of 0.14 in d'.
  • The authors observed that temporal precision alone (comparing Diffuse-67k to Weak-67k) contributed to a 0.11 d' improvement on the strong evaluation set.
  • The study found that using strong labels generally improved both d' and lwlrap metrics, though the optimal mixing ratio (μ) varied depending on the metric being optimized.

Limitations and Future Work

  • Label Interpretation Differences: The authors acknowledge that differences in label interpretation between the original weak labels and the new strong labels can create a disadvantage when evaluating models trained on one set using the other. They suggest automatic canonicalization of labels as a potential future direction.
  • Limited Scope of Strong Labels: The strong labels were collected for only a fraction (approximately 4%) of the original training data due to annotation costs.
  • Future Work: The authors propose exploring the interaction of annotator-based strong labels with automatic methods for weak label mitigation (e.g., autopooling). They also suggest that strong labels open up possibilities for models that directly predict segment boundaries, rather than relying on fixed-size frames.

Summary updated

BibTeX
@inproceedings{hershey2021benefit,
  author = {Shawn Hershey and Daniel P. W. Ellis and Eduardo Fonseca and Aren Jansen and Caroline Liu and R. Channing Moore and Manoj Plakal},
  title = {{The Benefit of Temporally-Strong Labels in Audio Event Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414579},
}
Figure