Fully Few-shot Class-incremental Audio Classification Using Multi-level Embedding Extractor and Ridge Regression Classifier

Yongjie Si, Yanxiong Li, Jiaxin Tan, Qianhua He, Il‐Youp Kwak2025 · DOI

TL;DR

This paper tackles the challenging problem of "Fully Few-shot Class-incremental Audio Classification" (FFCAC), where models must learn new audio classes with very few examples while retaining knowledge of previously learned classes. They propose a novel method that decouples audio feature extraction using a multi-level embedding extractor (MEE) from classification using a ridge regression classifier (RRC), achieving state-of-the-art performance with improved generalization.

What Problem Does This Paper Address?

Audio classification is crucial for many applications, but training robust models often requires abundant labeled data. The field has seen progress in incremental learning, where models learn new classes over time. However, existing "Few-shot Class-incremental Audio Classification" (FCAC) methods still demand a substantial number of training samples for the initial "base" classes. This paper addresses a more realistic scenario: "Fully FFCAC" (FFCAC), where both base and incremental classes have very limited training data. This scarcity poses a significant challenge, as models can easily forget old classes or fail to learn new ones effectively. Furthermore, existing FFCAC methods, while addressing data scarcity, often suffer from high computational complexity and suboptimal accuracy.

Methods

The proposed FFCAC method is built upon two key novel components:

  • Multi-level Embedding Extractor (MEE): This component is designed to capture rich and discriminative audio features. It utilizes a pre-trained Audio Spectrogram Transformer (AST) encoder, which processes audio spectrograms. Crucially, the MEE fuses features extracted from multiple layers of the AST encoder. The intuition is that shallower layers capture local, fine-grained details, while deeper layers capture more abstract, global characteristics. By combining these multi-level features, the MEE aims to create embeddings that are more representative and better distinguish between classes. The MEE is trained during the base session and then frozen for all subsequent incremental sessions.
  • Ridge Regression Classifier (RRC): To address the overfitting issue common in few-shot learning scenarios, the paper employs an RRC. Ridge regression is a technique that adds L2 regularization to the standard least squares method, effectively reducing the correlations between different dimensions of the feature embeddings. This regularization helps to improve the model's generalization ability, especially when dealing with limited data. A key advantage of the RRC is that its weights can be updated analytically, avoiding computationally expensive retraining in each incremental session. The RRC weights are updated using matrices derived from the embeddings and labels of previous sessions.

The overall framework involves training the MEE in the base session and then continually updating the RRC in each incremental session, while keeping the MEE fixed.

Datasets

The proposed method was evaluated on three public datasets commonly used for audio classification tasks:

  • LS-100
  • FSC-89
  • NSynth-100

These datasets were used in a class-incremental setting, where samples were split across multiple sessions, with each new session introducing new classes. The paper notes that the key difference between FCAC and FFCAC lies in the size of the base dataset; FFCAC assumes a small base dataset, which is the scenario tested here.

Key Insights and Ideas

  • Multi-level feature fusion is crucial for discriminative embeddings: By combining features from different depths of the AST encoder, the MEE can extract richer representations that are more effective for classification, especially in low-data regimes.
  • Ridge Regression combats overfitting in few-shot incremental learning: The L2 regularization inherent in RRC helps to mitigate the problem of feature dimension correlations, leading to better generalization when learning new classes with minimal examples.
  • Decoupling feature extraction and classification offers efficiency: Freezing the MEE after initial training and using an analytical update for the RRC significantly reduces computational burden during incremental learning compared to methods that retrain the entire model.
  • The proposed method achieves a good balance between accuracy and complexity: While not the absolute simplest, the method offers a competitive trade-off, outperforming many complex models in accuracy while remaining manageable in terms of computational resources.

Key Results

The proposed FFCAC method demonstrated superior performance across the evaluated datasets:

  • State-of-the-art Accuracy: The method achieved the highest Average Accuracy (AA) and the lowest Performance Degradation (PD) rates compared to several existing methods on LS-100, NSynth-100, and FSC-89. For instance, on LS-100, the proposed method achieved an AA of 57.36% and a PD of 37.14%, outperforming the next best method (AISP) by a significant margin.
  • Ablation Study Validation: Experiments confirmed that both the multi-level fusion in the MEE and the use of the RRC classifier contribute positively to the overall performance. The combination of both components yielded the best results.
  • Competitive Complexity: While having higher computational complexity (MACs) and parameter counts (NP) than simpler methods like Finetune and iCaRL, the proposed method's complexity is in the middle range, outperforming more complex attention-based methods in terms of efficiency while achieving better accuracy.

Limitations and Future Work

  • Complexity: The paper acknowledges that the MACs and NP of their method are higher than some baseline methods like Finetune and iCaRL. This is attributed to the complexity of the fusion module within the MEE.
  • Future Directions: The authors plan to further improve the accuracy and reduce the complexity of their method. They suggest exploring techniques such as model distillation and compression as potential avenues for future work.

Summary updated

BibTeX
@misc{si2025fully,
  author = {Yongjie Si and Yanxiong Li and Jiaxin Tan and Qianhua He and Il‐Youp Kwak},
  title = {{Fully Few-shot Class-incremental Audio Classification Using Multi-level Embedding Extractor and Ridge Regression Classifier}},
  year = {2025},
  doi = {10.21437/interspeech.2025-1085},
}
Figure