Learning from Taxonomy: Multi-label Few-Shot Classification for Everyday Sound Recognition

Jinhua Liang, Huy Phan, Emmanouil BenetosICASSP 2022 · OpenReview

TL;DR

This paper tackles the challenge of recognizing everyday sounds when only a few examples are available for each sound category (few-shot learning). They introduce a new dataset and a novel method, LaD-Protonet, that leverages the hierarchical structure (taxonomy) of sound events to improve classification accuracy, outperforming existing approaches.

What Problem Does This Paper Address?

Recognizing everyday sounds is crucial for many applications, but training accurate models typically requires vast amounts of labeled data. This becomes a bottleneck when dealing with rare sounds or when new sound categories emerge. Few-shot learning aims to address this by enabling models to learn from just a handful of examples. However, existing few-shot audio classification methods often treat sound categories as independent labels, ignoring the rich semantic relationships between them (e.g., "dog" is a type of "animal"). This paper addresses the limitations of current few-shot audio classification by proposing a method that explicitly incorporates audio taxonomy to improve recognition performance, especially in multi-label scenarios where multiple sounds can occur simultaneously.

Methods

The core of this work lies in two novel contributions:

  • Label-Dependent Prototypical Networks (LaD-Protonet): This method adapts the concept of prototypical networks, which learn by comparing query samples to class prototypes (average embeddings of support samples). LaD-Protonet transforms multi-label classification into multiple single-label tasks by considering each label associated with a query. Crucially, it leverages the parent-child relationships within the audio taxonomy. When a query has both a child and parent label, LaD-Protonet aligns these tasks, effectively forcing the model to learn more abstract concepts (parent classes) better than their specific instances (child classes). This is achieved by adjusting the support sets to include parent class information.
  • Taxonomy-Aware Label Smoothing: This technique enhances the training process by incorporating the audio taxonomy into the ground truth labels. Instead of using strict one-hot encodings (0 or 1), it assigns nuanced probabilities to negative classes based on their "distance" within the taxonomy. This encourages the model to be less overconfident and to better understand the relationships between classes, even those not directly present in the few-shot task.

Datasets

The authors introduce a new dataset called FSD-FS (Few-Shot). This dataset is curated from the existing FSD50K dataset and is designed for multi-label few-shot audio classification. It features 143 classes organized under the AudioSet taxonomy. Key characteristics of FSD-FS include:

  • Class Diversity: A broad range of sound events for generic audio classification research.
  • Polyphonic Sound Events: Examples can contain multiple sound events simultaneously, reflecting real-world scenarios.
  • Open Resource: Publicly available to facilitate reproducible research.

The dataset is split into base, validation, and evaluation sets, with specific strategies to ensure data integrity and prevent label leakage.

Key Insights and Ideas

  • Taxonomy is Informative: The paper strongly suggests that explicitly modeling the hierarchical structure of sound events significantly boosts few-shot classification performance. Ignoring these relationships is a missed opportunity.
  • Coarse-to-Fine Learning: LaD-Protonet's ability to align parent and child class tasks demonstrates the power of learning general concepts before specific ones, a principle often seen in human learning.
  • Softening Labels for Better Generalization: Taxonomy-aware label smoothing, by introducing nuanced probabilities, helps models generalize better and avoid overfitting to the limited few-shot examples.
  • Multi-label Few-Shot is Challenging: The authors highlight that standard "N-way K-shot" setups are not directly applicable to multi-label scenarios, necessitating novel task formation strategies.

Key Results

The proposed LaD-Protonet, especially when combined with taxonomy-aware label smoothing, consistently outperforms baseline methods and the "one-vs-rest" approach on the FSD-FS dataset. Specifically:

  • LaD-Protonet achieved a 1.34% improvement in mAP over non-ontology-based methods on the evaluation set.
  • The taxonomy-aware label smoothing technique, with an optimal parameter $\beta=30$, further boosted performance, indicating the benefit of embedding taxonomic knowledge.
  • The paper demonstrates that while higher $\beta$ values in label smoothing can improve validation performance, they might lead to overfitting on the evaluation set.

Limitations and Future Work

The authors acknowledge that their work, while addressing few-shot learning without forgetting, is not a full solution to that problem. They suggest several promising future directions:

  • Expanding FSD-FS: Further extending the dataset's label set, diversifying its data distribution, and refining the audio taxonomy based on physical characteristics.
  • Leveraging Label Correlation: Exploring methods to utilize label correlations without relying on a predefined taxonomy.
  • Physical Characteristics: Investigating how to incorporate physical properties of sound sources into the classification process.

Summary updated

Abstract

Everyday sound recognition aims to infer types of sound events in audio streams. While many works succeeded in training models with high performance in a fully-supervised manner, they are still restricted to the demand of large quantities of labelled data and the range of predefined classes. To overcome these drawbacks, this work firstly curates a new database named FSD-FS for multi-label few-shot audio classification. It then explores how to incorporate audio taxonomy in few-shot learning. Specifically, this work proposes label-dependent prototypical networks (LaD-protonet) to exploit parent-children relationships between labels. Plus, it applies taxonomy-aware label smoothing techniques to boost model performance. Experiments demonstrate that LaD-protonet outperforms original prototypical networks as well as other state-of-the-art methods. Moreover, its performance can be further boosted when combined with taxonomy-aware label smoothing.
BibTeX
@inproceedings{liang2022learning,
  author = {Jinhua Liang and Huy Phan and Emmanouil Benetos},
  title = {{Learning from Taxonomy: Multi-label Few-Shot Classification for Everyday Sound Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=XBzUYG7jSW},
}
Figure