Leveraging Label Hierachies for Few-Shot Everyday Sound Recognition

Jinhua Liang, Huy Phan, Emmanouil BenetosDCASE 2022 · OpenReview

Abstract

Recognizing everyday environmental sounds remains a significant challenge in machine learning due to the high diversity of audio events and the limited availability of labeled datasets. This paper introduces Hierarchical Prototypical Networks (HPNs), a few-shot learning (FSL) framework that incorporates structured audio taxonomies into the training process. By organizing the embedding space to reflect semantic relationships between sounds, HPNs achieve state-of-the-art performance on the ESC-50 dataset, demonstrating that hierarchical knowledge is a powerful prior for learning from minimal examples.

Introduction

From the hum of a refrigerator to the chirp of a cricket, everyday sounds are ubiquitous yet difficult for AI to categorize reliably. While Few-Shot Learning (FSL) has emerged as a solution for recognizing new classes with only a few labeled examples (the "N-way K-shot" problem), most FSL algorithms treat labels as independent, flat categories. However, sounds naturally exist within hierarchies—for instance, "Dogs" and "Cats" are both "Animals," while "Chainsaws" and "Engines" fall under "Exterior/urban noises."

Humans leverage this structural knowledge to generalize quickly. This paper bridges the gap by integrating these taxonomies directly into the learning objective, forcing the model to understand not just what a sound is, but where it fits in the broader family tree of audio.

Methodology: Hierarchical Prototypical Networks

The proposed HPN architecture builds upon the standard Prototypical Network by introducing a multi-level embedding strategy.

Feature Extraction and Prototypes

The model utilizes an 8-layer VGG-like convolutional neural network (CNN) to transform log Mel spectrograms into 256-dimensional embeddings. In a standard Prototypical Network, a "prototype" is defined as the mean vector of all support examples for a specific class. HPNs extend this by calculating prototypes at every level of the hierarchy:

  1. Bottom Level: Prototypes are calculated for specific child classes (e.g., "Pig").
  2. Higher Levels: These specific prototypes are aggregated to form "parent" prototypes (e.g., "Animals"), representing the centroid of a broader category.
Conceptual diagram of Hierarchical Prototypical Networks (HPNs) mapping child classes to parent categories.

Multi-Task Learning and Structural Loss

To enforce the hierarchy, the model is trained to solve classification tasks at every level simultaneously. A weighted multi-level cross-entropy loss—referred to as Structural Loss—is applied. This ensures that the embedding space is organized such that "sibling" classes are positioned closer to each other than to "distant cousins," effectively using the taxonomy as a regularizer.

Experimental Setup

The researchers evaluated HPN on the ESC-50 dataset, which consists of 2,000 five-second recordings across 50 classes. These classes are grouped into five major parent categories: Animals, Natural & water soundscapes, Human non-speech sounds, Interior/domestic sounds, and Exterior/urban noises.

A key finding in the experimental setup was the importance of the data split. The researchers found that a "Uniform Split"—where the ratio of child classes under each parent remains consistent between training and testing—yielded the best results. This suggests that exposure to related "cousin" classes during training significantly aids the model's ability to generalize to new "sibling" classes during few-shot evaluation.

Results and Discussion

HPN consistently outperformed baseline few-shot methods, including standard Prototypical Networks, Matching Networks, and Transfer Learning approaches.

Performance Benchmarks

In the challenging 12-way 5-shot classification task, HPN achieved an accuracy of 78.65%, a notable improvement over the 77.70% achieved by standard Prototypical Networks. Similar gains were observed in 5-way 5-shot tasks and across F1-scores, indicating a robust balance between precision and recall.

Performance comparison on the 12-way 5-shot classification task using the ESC-50 dataset.

The results confirm that a priori taxonomic knowledge acts as a powerful guide for the encoder, helping it learn embeddings that reflect semantic similarity rather than just raw acoustic features.

Conclusion and Future Work

While HPNs show great promise, they currently rely on predefined, explicit taxonomies and tree-based structures. Future work aims to address more complex "multi-path" taxonomies where a sound might belong to multiple categories. Additionally, the researchers plan to scale this approach to larger datasets like AudioSet and explore "knowledge-discovery" methods that can automatically infer hierarchical relationships without manual labeling.

The source code and evaluation environment are available on GitHub.

Summary updated

BibTeX
@inproceedings{liang2022leveraging,
  author = {Jinhua Liang and Huy Phan and Emmanouil Benetos},
  title = {{Leveraging Label Hierachies for Few-Shot Everyday Sound Recognition}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=Mdb5kZaXmG},
}
Figure