Abstract
Few-Shot Class Incremental Learning (FSCIL) requires models to learn new categories from minimal data without forgetting previous knowledge. Current methods often struggle with the stability-plasticity dilemma, particularly in distinguishing fine-grained classes when the backbone is frozen. We propose FineFMPL (Fine-grained Feature Mining Prompt Learning), a framework that leverages Vision-Language models to extract discriminative object-level features. By introducing Visual Probe Prompts (VPP) and cross-modal mapping to create robust visual and textual prototypes, FineFMPL achieves state-of-the-art performance on benchmarks like CUB-200-2011, CIFAR100, and miniImageNet, significantly reducing catastrophic forgetting.
The Challenge of Incremental Learning
In machine learning, the "Stability-Plasticity" dilemma describes the difficulty of creating models that are plastic enough to learn new information but stable enough to retain old knowledge. This is particularly acute in Few-Shot Class Incremental Learning (FSCIL), where a model must learn new classes from as few as 5 or 10 examples after being trained on a large base dataset.
Most existing FSCIL methods attempt to prevent "catastrophic forgetting" by freezing the model's feature extractor (backbone). However, a frozen backbone often fails to capture the unique, fine-grained details necessary to distinguish between similar new classes, such as different species of birds.
Methodology: Fine-Grained Feature Mining
FineFMPL addresses these challenges by leveraging the pre-trained capabilities of Vision-Language (VL) models like CLIP. The framework focuses on mining discriminative information through three primary components:
- Visual Probe Prompt (VPP): A learnable "probe" is inserted into the image encoder to interact with image patches. This mechanism identifies significant regions (e.g., a bird's beak or wing pattern) and filters out irrelevant background noise.
- Dual Visual Prototypes: For every class, the model generates a Global-level Prototype (overall appearance) and an Object-level Prototype (fine-grained details identified by the VPP).
- Textual Context Prompting: To create a redundant memory system, visual prototypes are mapped into the textual domain. These "Textual Prototypes" provide a second, non-visual anchor for each class, making the model's memory more resilient to weight shifts during incremental sessions.
Experimental Evaluation
FineFMPL was evaluated on three standard FSCIL benchmarks: CUB-200-2011 (fine-grained birds), CIFAR100, and miniImageNet.
On the challenging CUB-200-2011 dataset, FineFMPL achieved a state-of-the-art average accuracy of 79.7% across all sessions. This significantly outperformed previous CNN-based methods like SAVC (69.4%) and recent prompt-based methods like IOS (71.0%). Notably, FineFMPL maintained a high accuracy of 76.4% even in the final session after learning 100 new classes, demonstrating exceptional resistance to performance decay.
Conclusion
FineFMPL demonstrates that focusing on fine-grained details and utilizing cross-modal redundancy are key to solving the stability-plasticity dilemma in few-shot learning. By effectively mining object-level features and anchoring them in both visual and textual spaces, the framework provides a robust solution for continuous learning in complex, real-world scenarios.
Summary updated