TL;DR
This paper tackles the challenging problem of "fully few-shot class-incremental audio classification" (FFCAC), where models must learn new audio classes with very few examples per class, across multiple learning sessions, without forgetting previously learned classes. They introduce an "Expandable Dual-embedding Extractor" (EDE) that leverages both a pre-trained and a fine-tuned Audio Spectrogram Transformer (AST) to create more robust and adaptable audio representations, achieving state-of-the-art results on challenging benchmarks.
What Problem Does This Paper Address?
Traditional audio classification models often assume ample training data is available, especially for initial learning. However, many real-world scenarios, like identifying rare animal calls or specific criminal voices, suffer from extreme data scarcity. Furthermore, in class-incremental learning, models need to continuously learn new classes without forgetting old ones. Existing "few-shot class-incremental" (FCAC) methods still require a substantial amount of data in the initial "base session" to train a good embedding extractor. This paper addresses the even more challenging "fully few-shot class-incremental audio classification" (FFCAC) setting, where all sessions, including the base session, have very limited training samples. This scarcity severely impacts both the model's ability to retain old knowledge (stability) and its capacity to learn new information (plasticity).
Methods
The core of the proposed solution is the Expandable Dual-embedding Extractor (EDE). The EDE builds upon the Audio Spectrogram Transformer (AST), a powerful attention-based model for audio classification. Here's how it works:
- Dual-embedding: The EDE combines two AST components:
- A pretrained AST (pre) that captures general audio features.
-
A fine-tuned AST (ψ₀) trained on the limited data of the base session. This dual approach aims to leverage the generalizability of the pretrained model and the task-specific adaptation of the fine-tuned model, creating complementary embeddings.
-
Expandable Classifier: To further enhance plasticity, the EDE dynamically expands. In each incremental session, the deep layers of the AST are extended, while earlier layers are frozen. This allows the model to learn new, task-specific embeddings without disrupting the general knowledge. Prototypes for each class are then computed as the mean of these embeddings.
- Prototype-based Classification: The classifier is composed of these learned prototypes, representing both old and new classes.
- Reconstruction of Old Embeddings: To combat forgetting, the paper also proposes reconstructing embeddings of old classes using their saved covariance matrices, helping to reinforce past knowledge.
Datasets
The experiments were conducted on three publicly available datasets:
- LS-100: A dataset for audio classification.
- NSynth-100: A dataset focusing on musical instrument sounds.
- FSC-89: Another dataset for audio classification.
For their experiments, the authors randomly selected 25 classes from each dataset and split them into multiple sessions (M sessions) for incremental learning. The specific details of these datasets (number of classes, duration, samples per class) are provided in Table 1.
Key Insights and Ideas
- The "Fully Few-Shot" Challenge is Real: The paper highlights that even existing few-shot incremental methods struggle when the base session also has very limited data, underscoring the difficulty of FFCAC.
- Dual Embeddings for Robustness: Combining a general pretrained model with a task-specific fine-tuned model is a powerful strategy for improving both stability and plasticity in incremental learning.
- Dynamic Expansion is Key: Allowing the model to expand its capacity in deeper layers for new tasks, while keeping earlier layers fixed, is crucial for adapting to new information without catastrophic forgetting.
- Prototypes as Class Representatives: Using mean embeddings as prototypes is an effective way to represent classes in a few-shot setting.
Key Results
The proposed EDE method demonstrated superior performance across all three datasets compared to seven baseline methods.
- Average Accuracy (AA): The EDE achieved the highest AA scores, specifically:
-
LS-100: 61.85%
- NSynth-100: 65.40%
- FSC-89: 37.36% These results were statistically significant, indicating a genuine advantage over the baselines.
-
Ablation Studies: Experiments showed that using both the pretrained AST and the expanded fine-tuned AST (the full EDE) yielded the best results, confirming the effectiveness of the dual-embedding and expansion strategies.
Limitations and Future Work
- Memory Overhead: The authors acknowledge that the EDE, particularly the AST backbone and its expansion, requires a significant amount of memory. This could be a bottleneck for deployment on resource-constrained devices.
- Future Work: The primary direction for future work is to reduce the memory overhead of the EDE, making it more practical for real-world applications.
Summary updated