Fully Few-shot Class-incremental Audio Classification Using Expandable Dual-embedding Extractor

Yongjie Si, Yanxiong Li, Jialong Li, Jiaxin Tan, Qianhua He

TL;DR

This paper introduces "Fully Few-shot Class-incremental Audio Classification" (FFCAC), a challenging scenario where models must learn new audio classes with very few samples, even during the initial training phase. To tackle this, they propose an "Expandable Dual-embedding Extractor" (EDE) that leverages both a pre-trained and a fine-tuned Audio Spectrogram Transformer (AST) to create more robust and adaptable audio representations, outperforming existing methods.

What Problem Does This Paper Address?

Traditional audio classification models often assume ample data is available for initial training. However, real-world applications like identifying rare animal sounds or specialized industrial noises face severe data scarcity, even for the "base" or initial set of classes. Furthermore, these models need to continuously learn new classes over time without forgetting previously learned ones (class-incremental learning) and with minimal new data (few-shot learning). Existing "Few-shot Class-incremental Audio Classification" (FCAC) methods still require a substantial amount of data for the initial training session to build a good embedding extractor. This paper addresses the even more challenging "Fully Few-shot Class-incremental Audio Classification" (FFCAC) problem, where all training sessions, including the base session, have very limited data. This lack of initial data severely hampers the model's ability to learn stable and adaptable representations, leading to poor performance in both remembering old classes and learning new ones.

Methods

The core of the proposed solution is the Expandable Dual-embedding Extractor (EDE). The EDE aims to overcome the limitations of using either a pre-trained or a fine-tuned Audio Spectrogram Transformer (AST) in isolation.

  • Dual-embedding Extractor: The EDE combines two ASTs:
  • Pre-trained AST ($\psi_{pre}$): This AST, trained on large, general audio datasets, provides robust, task-general embeddings.
  • Fine-tuned AST ($\psi_0$): This AST is fine-tuned on the limited data of the current session. While it captures task-specific features, it can be prone to forgetting old classes. By merging these two, the EDE leverages the strengths of both, creating complementary embeddings that improve model stability.

  • Expandable Classifier: To further enhance the model's ability to learn new classes (plasticity), the EDE is dynamically expanded in each incremental session. Specifically, the deep layers of the AST are extended, while the earlier, generalizable layers are frozen. This allows the model to learn new, task-specific embeddings for each new session. Prototypes for each class are then computed as the mean of these extracted embeddings, forming the classifier.

  • Reconstruction of Old Embeddings: To mitigate forgetting, the paper also proposes reconstructing embeddings of old classes using their saved covariance matrices and a standard normal distribution. This helps to reinforce the representations of previously learned classes.

Datasets

The experiments were conducted on three publicly available datasets:

  • LS-100: Contains 100 classes of audio.
  • NSynth-100: Contains 100 classes of musical instrument sounds.
  • FSC-89: Contains 89 classes of audio.

For each dataset, the authors randomly selected 25 classes and split them into multiple sessions (base and incremental) with a "N-way K-shot" setting (specifically N=5, K=5) for training, meaning each new class in an incremental session is represented by only 5 examples per class.

Key Insights and Ideas

  • The "Fully Few-Shot" Challenge is Real: The paper highlights that even the initial training phase can suffer from extreme data scarcity, a problem not adequately addressed by prior FCAC methods.
  • Complementarity is Key: Combining a general pre-trained model with a task-specific fine-tuned model is crucial for robust representation learning in low-data regimes. The dual-embedding approach effectively captures both general and specific audio characteristics.
  • Dynamic Expansion for Plasticity: Allowing the model's feature extractor to grow and adapt to new classes, while preserving learned knowledge, is essential for effective incremental learning.
  • Reconstruction Aids Memory: Reconstructing old class embeddings provides a mechanism to combat catastrophic forgetting, reinforcing previously learned patterns.

Key Results

The proposed EDE method demonstrated superior performance across all three datasets compared to seven baseline methods.

  • Average Accuracy (AA): The EDE achieved the highest AA scores, specifically:
  • LS-100: 61.85%

    • NSynth-100: 65.40%
    • FSC-89: 37.36% These results were statistically significant, indicating a genuine advantage over the baselines.
  • Ablation Studies: Experiments showed that using both the pre-trained AST and the expanded fine-tuned AST (i.e., the full EDE) yielded the best results, confirming the effectiveness of each component.

Limitations and Future Work

  • Memory Overhead: The authors acknowledge that the EDE, being based on a pre-trained AST and expanding over time, requires a significant amount of memory. This memory requirement increases with each incremental session.
  • Future Work: The primary direction for future research is to reduce the memory overhead of the EDE. This suggests exploring more memory-efficient architectures or compression techniques for the expandable components.

Summary updated

Abstract

It's assumed that training data is sufficient in base session of few-shot class-incremental audio classification. However, it's difficult to collect abundant samples for model training in base session in some practical scenarios due to the data scarcity of some classes. This paper explores a new problem of fully few-shot class-incremental audio classification with few training samples in all sessions. Moreover, we propose a method using expandable dual-embedding extractor to solve it. The proposed model consists of an embedding extractor and an expandable classifier. The embedding extractor consists of a pretrained Audio Spectrogram Transformer (AST) and a finetuned AST. The expandable classifier consists of prototypes and each prototype represents a class. Experiments are conducted on three datasets (LS-100, NSynth-100 and FSC-89). Results show that our method exceeds seven baseline ones in average accuracy with statistical significance. Code is at: https://github.com/YongjieSi/EDE.
BibTeX
@misc{si2024fully,
  author = {Yongjie Si and Yanxiong Li and Jialong Li and Jiaxin Tan and Qianhua He},
  title = {{Fully Few-shot Class-incremental Audio Classification Using Expandable Dual-embedding Extractor}},
  year = {2024},
  eprint = {2406.08122},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.08122},
}
Figure