Training neural audio classifiers with few data

Jordi Pons, Joan Serrà, Xavier Serra

Abstract: We investigate supervised learning strategies that improve the training of neural network audio classifiers on small annotated collections. In particular, we study whether (i) a naive regularization of the solution space, (ii) prototypical networks, (iii) transfer learning, or (iv) their combination, can foster deep learning models to better leverage a small amount of training examples. To this end, we evaluate (i-iv) for the tasks of acoustic event recognition and acoustic scene classification, considering from 1 to 100 labeled examples per class. Results indicate that transfer learning is a powerful strategy in such scenarios, but prototypical networks show promising results when one does not count with external or validation data.

Introduction

Deep learning has revolutionized "machine listening," yet it remains notoriously data-hungry. Most state-of-the-art audio classifiers rely on datasets where every category contains hundreds or thousands of examples. However, in many real-world applications, labeling audio is a tedious and expensive process, leading to "low-data regimes" where only 5 or 10 examples of a specific acoustic event may be available. This research addresses a fundamental question: How can neural networks be designed to learn more like humans, extracting meaningful patterns from just a few examples?

Methodology and Datasets

The study evaluates four distinct strategies for handling data scarcity:

  1. Regularized Baselines: Standard VGG-style networks and a specialized, low-parameter model called TIMBRE, which uses domain-specific vertical filters to capture frequency signatures.
  2. Prototypical Networks: A few-shot learning architecture that learns an embedding space where examples of the same class cluster together around a calculated "prototype" (mean vector).
  3. Transfer Learning: Fine-tuning a large VGG model pre-trained on AudioSet (2 million YouTube clips).
  4. Hybrid Approaches: Combinations of transfer learning with prototypical architectures.

Experiments were conducted on two benchmarks: UrbanSound8K (US8K) for acoustic event recognition and TUT Acoustic Scenes (ASC-TUT) for environmental scene classification. Crucially, the authors assumed a realistic scenario where no validation set exists, necessitating robust early-stopping rules.

The "Just Overfit" Heuristic

A significant finding of this work is the counter-intuitive stability of Prototypical Networks. While standard networks typically suffer from catastrophic overfitting when trained on tiny datasets, Prototypical Nets exhibit remarkable resilience. The researchers discovered that simply training until the model perfectly classifies the training set (100% accuracy) often leads to optimal generalization on the test set, even without a validation set for early stopping.

Train vs Test accuracy over epochs, illustrating the stability of Prototypical Networks.

Comparative Results and Analysis

The effectiveness of these strategies varies significantly based on domain alignment and the number of training examples ($n$).

  • Transfer Learning Dominance: On the US8K dataset, transfer learning was the clear leader. With $n=100$, it achieved approximately 74% accuracy, far surpassing models trained from scratch. This success is attributed to the similarity between the source domain (AudioSet events) and the target domain (urban events).
  • Prototypical Net Resilience: On the ASC-TUT dataset, Prototypical Networks trained from scratch proved highly competitive. As $n$ exceeded 20, they often outperformed transfer learning from AudioSet. This suggests that when the pre-trained model comes from a different domain (e.g., events vs. scenes), metric-learning approaches like Prototypical Nets provide a safer, more robust alternative.
  • Architecture as Regularization: For extremely small $n$ (less than 10), the tiny TIMBRE model frequently outperformed larger, more expressive networks, demonstrating that restricted model capacity can serve as a powerful form of regularization when data is scarce.
Accuracy vs. training examples (n) for various models on US8K and ASC-TUT.

Conclusion

While transfer learning remains a powerhouse when relevant source data is available, its success is highly dependent on domain alignment. In niche applications where massive pre-trained models do not exist—such as underwater bioacoustics or industrial monitoring—the "from-scratch" prototypical approach offers the most promising path. Future research will focus on more sophisticated distance metrics and bridging the representation gap between acoustic events and environmental scenes.

Summary updated

Abstract

We investigate supervised learning strategies that improve the training of neural network audio classifiers on small annotated collections. In particular, we study whether (i) a naive regularization of the solution space, (ii) prototypical networks, (iii) transfer learning, or (iv) their combination, can foster deep learning models to better leverage a small amount of training examples. To this end, we evaluate (i-iv) for the tasks of acoustic event recognition and acoustic scene classification, considering from 1 to 100 labeled examples per class. Results indicate that transfer learning is a powerful strategy in such scenarios, but prototypical networks show promising results when one does not count with external or validation data.
BibTeX
@inproceedings{pons2019training,
  author = {Jordi Pons and Joan Serrà and Xavier Serra},
  title = {{Training neural audio classifiers with few data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=jouo1RckTn},
}
Figure