Abstract
Few-Shot Class Incremental Learning (FSCIL) aims to continually learn new classes with few training samples without forgetting already learned old classes. Existing FSCIL methods generally fix the backbone network in incremental sessions to achieve a balance between suppressing forgetting old classes and learning new classes. However, the fixed backbone network causes insufficient learning of new classes from a few samples. Benefiting from the powerful visual and textual understanding ability of Vision-Language (VL) pre-training models, we propose a Fine-grained Feature Mining Prompt Learning (FineFMPL) approach to adapt the VL model to FSCIL, which comprehensively learns and memorizes fine-grained discriminative information of emerging classes. Concretely, the visual probe prompt is firstly proposed to guide the image encoder of VL model to extract global-level coarse-grained features and object-level fine-grained features, and visual prototypes are preserved based on image patch significance, which contains the discriminative characteristics exclusive to the class. Secondly, the textual context prompt is constructed by cross-modal mapping of visual prototypes, feeding into the text encoder of VL model to memorize the class information as textual prototypes. Finally, integrating visual and textual prototypes based on fine-grained feature mining into the model improves the recognition performance of all classes in FSCIL. Extensive experiments on three benchmark datasets demonstrate that our FineFMPL achieves new state-of-the-art.
Introduction
In the domain of machine learning, Few-Shot Class Incremental Learning (FSCIL) represents a significant frontier. It requires models to learn new classes from a handful of examples while maintaining performance on previously learned "base" classes. This task is governed by the Stability-Plasticity Dilemma: the need to remain stable enough to prevent catastrophic forgetting of old knowledge, while being plastic enough to incorporate new information from limited data.
Traditional FSCIL approaches often freeze the feature extractor after the base session to ensure stability. However, this limits the model's ability to learn the subtle, discriminative features necessary for fine-grained classification of new species or objects. FineFMPL addresses this by adapting large-scale Vision-Language (VL) models, such as CLIP, to mine and memorize these fine-grained details through a novel prompting mechanism.
Methodology
The FineFMPL framework leverages the dual-modality strengths of VL models to create a robust memory of both global and local class features.
Visual Probe Prompting (VPP)
To extract discriminative information, FineFMPL introduces a Visual Probe Prompt (VPP). This learnable token is inserted into the image encoder to "probe" and gather information from significant image patches. This process allows the model to generate two types of features: coarse-grained global features and fine-grained object features. By focusing on patch significance identified through attention maps, the model isolates the most relevant parts of an image (e.g., a bird's beak or wing pattern) to form visual prototypes.
Textual Context Prompting (TCP)
To further solidify the model's memory and alleviate forgetting, FineFMPL employs Textual Context Prompting (TCP). Visual prototypes are mapped into the textual embedding space via a cross-modal mapping module. These embeddings serve as context for the text encoder, generating textual prototypes—essentially high-dimensional digital descriptions of the class. This dual-modality approach ensures that class information is stored both visually and textually, providing redundancy that aids classification during incremental sessions.
Inference and Matching
During inference, a two-pathway matching strategy is used. The input image's features are compared against both the stored visual prototypes and the generated textual prototypes. This integrated approach allows the model to make more accurate predictions by considering both global context and fine-grained details.
Experiments and Results
FineFMPL was evaluated on three standard FSCIL benchmarks: CUB-200-2011 (fine-grained birds), CIFAR100 (general objects), and miniImageNet. In all cases, the model followed the standard FSCIL setup: a base session with ample data followed by incremental sessions with only 5 samples per new class.
- CUB-200-2011: On this challenging fine-grained bird dataset, FineFMPL achieved an average accuracy of 79.7%, outperforming the previous state-of-the-art (IOS) by 8.7%.
- CIFAR100 & miniImageNet: The model demonstrated strong generalization, achieving average accuracies of 84.2% and 93.4%, respectively.
The experimental results indicate that FineFMPL significantly flattens the accuracy drop curve typically seen in incremental learning, demonstrating superior stability without sacrificing the plasticity needed for few-shot learning.
Conclusion
FineFMPL introduces a powerful method for FSCIL by mining fine-grained features through visual and textual prompting. By steering pre-trained Vision-Language models to focus on discriminative details, it effectively overcomes the limitations of frozen backbones. This approach sets a new state-of-the-art across multiple benchmarks, highlighting the potential of cross-modal knowledge utilization in continual learning scenarios.
Summary updated