Semi-Supervised Audio Classification with Partially Labeled Data

Siddharth Gururani, Alexander Lerch

Introduction

Training robust audio classifiers on large-scale datasets is frequently complicated by the phenomenon of "partial labeling." Unlike image annotation, where objects can be identified at a glance, audio annotation requires human listeners to spend significant time—often multiple passes—to identify every sound present in a clip. Consequently, many datasets are sparsely labeled: an annotator may confirm the presence of a "guitar" but leave other instruments like "drums" or "piano" unverified.

Comparison between fully labeled and partially labeled audio data.

A common but detrimental practice in the field is to treat these missing labels as negative instances (absent). This research investigates how this "missing-is-negative" assumption introduces false negatives that degrade model performance and explores semi-supervised learning (SSL) techniques to mitigate these effects.

Methodology

The study evaluates four distinct approaches to handling missing labels:

  1. Baseline 0 (B0): The naive approach where all missing labels are treated as negative.
  2. Baseline 1 (B1): The "omission" approach, where missing labels are masked out and ignored during the loss calculation.
  3. Label Enhancing (LE): A two-stage teacher-student framework. A teacher model (trained via B0) identifies missing labels likely to be positive. These labels are then masked to create an "enhanced" set for training a student model.
  4. Mean Teacher (MT): A consistency-based SSL method. A Student network is trained to match the output of a Mean Teacher network (a moving average of the Student's weights), ensuring predictions remain stable even under input perturbations.

Experimental Setup

The methods were tested on two datasets with varying levels of sparsity:

  • OpenMIC-2018: A music dataset where approximately 90% of labels are missing.
  • SONYC-UST: An urban sound dataset with a much lower missing label rate of roughly 6%.

Models utilized pre-trained embeddings (VGGish for music and OpenL3 for urban sounds) processed through an attention-based neural network.

Key Results and Discussion

The experimental results highlight the inefficiency of the standard B0 approach. Simply ignoring missing labels (B1) proved to be a surprisingly strong baseline, suggesting that no information is often better than incorrect information.

Performance comparison (Micro and Macro AUPRC) across different levels of missing labels on the SONYC-UST dataset.

As shown in the performance metrics, the semi-supervised methods—particularly Mean Teacher and Label Enhancing—demonstrate superior robustness as label sparsity increases. In simulations where up to 80% of labels were intentionally removed from the SONYC-UST dataset, Label Enhancing maintained high performance while the naive baseline's accuracy plummeted. On the highly sparse OpenMIC dataset, the Mean Teacher method was the most effective, leveraging consistency regularization to learn the underlying audio structure despite the lack of explicit labels.

Conclusion and Future Work

This research demonstrates that semi-supervised techniques can effectively recover performance lost to partial labeling. While methods like Mean Teacher require metadata to identify which labels are missing, and Label Enhancing requires careful threshold tuning, both offer significant advantages over traditional training methods. Future research may focus on hybrid approaches that combine consistency regularization with multi-stage label enhancement to further refine training signals in sparse data environments.

Summary updated

Abstract

Audio classification has seen great progress with the increasing availability of large-scale datasets. These large datasets, however, are often only partially labeled as collecting full annotations is a tedious and expensive process. This paper presents two semi-supervised methods capable of learning with missing labels and evaluates them on two publicly available, partially labeled datasets. The first method relies on label enhancement by a two-stage teacher-student learning process, while the second method utilizes the mean teacher semi-supervised learning algorithm. Our results demonstrate the impact of improperly handling missing labels and compare the benefits of using different strategies leveraging data with few labels. Methods capable of learning with partially labeled data have the potential to improve models for audio classification by utilizing even larger amounts of data without the need for complete annotations.
BibTeX
@inproceedings{gururani2021semi,
  author = {Siddharth Gururani and Alexander Lerch},
  title = {{Semi-Supervised Audio Classification with Partially Labeled Data}},
  booktitle = {ISM},
  year = {2021},
  url = {https://openreview.net/forum?id=haiOy72waV},
}
Figure