FineFMPL: Fine-grained Feature Mining Prompt Learning for Few-Shot Class Incremental Learning

Hongbo Sun, Jiahuan Zhou, Xiangteng He, Jinglin Xu, Yuxin Peng, Ziteng Qiao, Yuxian Li, Jianqiang Xia

Abstract

Few-Shot Class Incremental Learning (FSCIL) requires models to learn new categories from minimal data without forgetting previous knowledge. Current methods often struggle with the stability-plasticity dilemma, particularly in distinguishing fine-grained classes when the backbone is frozen. We propose FineFMPL (Fine-grained Feature Mining Prompt Learning), a framework that leverages Vision-Language models to extract discriminative object-level features. By introducing Visual Probe Prompts (VPP) and cross-modal mapping to create robust visual and textual prototypes, FineFMPL achieves state-of-the-art performance on benchmarks like CUB-200-2011, CIFAR100, and miniImageNet, significantly reducing catastrophic forgetting.

The Challenge of Incremental Learning

In machine learning, the "Stability-Plasticity" dilemma describes the difficulty of creating models that are plastic enough to learn new information but stable enough to retain old knowledge. This is particularly acute in Few-Shot Class Incremental Learning (FSCIL), where a model must learn new classes from as few as 5 or 10 examples after being trained on a large base dataset.

Overview of the FSCIL task and the FineFMPL approach.

Most existing FSCIL methods attempt to prevent "catastrophic forgetting" by freezing the model's feature extractor (backbone). However, a frozen backbone often fails to capture the unique, fine-grained details necessary to distinguish between similar new classes, such as different species of birds.

Methodology: Fine-Grained Feature Mining

FineFMPL addresses these challenges by leveraging the pre-trained capabilities of Vision-Language (VL) models like CLIP. The framework focuses on mining discriminative information through three primary components:

  1. Visual Probe Prompt (VPP): A learnable "probe" is inserted into the image encoder to interact with image patches. This mechanism identifies significant regions (e.g., a bird's beak or wing pattern) and filters out irrelevant background noise.
  2. Dual Visual Prototypes: For every class, the model generates a Global-level Prototype (overall appearance) and an Object-level Prototype (fine-grained details identified by the VPP).
  3. Textual Context Prompting: To create a redundant memory system, visual prototypes are mapped into the textual domain. These "Textual Prototypes" provide a second, non-visual anchor for each class, making the model's memory more resilient to weight shifts during incremental sessions.
The FineFMPL architecture, highlighting Global Feature Learning, Visual Probe Prompting, and Textual Context Prompting.

Experimental Evaluation

FineFMPL was evaluated on three standard FSCIL benchmarks: CUB-200-2011 (fine-grained birds), CIFAR100, and miniImageNet.

On the challenging CUB-200-2011 dataset, FineFMPL achieved a state-of-the-art average accuracy of 79.7% across all sessions. This significantly outperformed previous CNN-based methods like SAVC (69.4%) and recent prompt-based methods like IOS (71.0%). Notably, FineFMPL maintained a high accuracy of 76.4% even in the final session after learning 100 new classes, demonstrating exceptional resistance to performance decay.

Conclusion

FineFMPL demonstrates that focusing on fine-grained details and utilizing cross-modal redundancy are key to solving the stability-plasticity dilemma in few-shot learning. By effectively mining object-level features and anchoring them in both visual and textual spaces, the framework provides a robust solution for continuous learning in complex, real-world scenarios.

Summary updated

BibTeX
@inproceedings{sun2024finefmpl,
  author = {Hongbo Sun and Jiahuan Zhou and Xiangteng He and Jinglin Xu and Yuxin Peng and Ziteng Qiao and Yuxian Li and Jianqiang Xia},
  title = {{FineFMPL: Fine-grained Feature Mining Prompt Learning for Few-Shot Class Incremental Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=u1nWcBVv5z},
}
Figure