MgSvF: Multi-Grained Slow vs. Fast Framework for Few-Shot Class-Incremental Learning

Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, Xi Li

Abstract

As a challenging problem, few-shot class-incremental learning (FSCIL) continually learns a sequence of tasks, confronting the dilemma between slow forgetting of old knowledge and fast adaptation to new knowledge. In this paper, we concentrate on this "slow vs. fast" (SvF) dilemma to determine which knowledge components to be updated in a slow fashion or a fast fashion, and thereby balance old-knowledge preservation and new-knowledge adaptation. We propose a multi-grained SvF learning strategy to cope with the SvF dilemma from two different grains: intra-space (within the same feature space) and inter-space (between two different feature spaces). The proposed strategy designs a novel frequency-aware regularization to boost the intra-space SvF capability, and meanwhile develops a new feature space composition operation to enhance the inter-space SvF learning performance. With the multi-grained SvF learning strategy, our method outperforms the state-of-the-art approaches by a large margin.

Introduction

Few-Shot Class-Incremental Learning (FSCIL) requires models to learn new classes from limited data while retaining knowledge of previously learned base classes. This task is plagued by the "Slow vs. Fast" (SvF) dilemma: the need for slow parameter updates to prevent catastrophic forgetting of old tasks versus the need for fast adaptation to learn new tasks effectively from sparse data.

The Slow vs. Fast (SvF) Dilemma

Traditional incremental learning methods often fall into two traps. "Slow forgetting" preserves old knowledge but fails to discriminate between new classes (underfitting). Conversely, "fast adaptation" allows the model to learn new classes quickly but leads to the loss of old knowledge (catastrophic forgetting). The ideal approach, SvF, balances these two extremes to maintain high performance across all tasks.

Conceptual comparison of slow forgetting, fast adaptation, and the proposed SvF approach.

Multi-Grained SvF Framework

The MgSvF framework addresses this dilemma through a multi-grained strategy operating at two levels: intra-space and inter-space.

Intra-space SvF: Frequency-Aware Regularization

The authors observed that different feature components exhibit varying degrees of stability. By applying a Discrete Cosine Transform (DCT), they found that low-frequency components are naturally more resistant to forgetting than high-frequency ones.

Empirical evidence showing that forgetting increases with frequency components over 10 incremental sessions.

Based on this insight, MgSvF employs frequency-aware regularization. It imposes a higher penalty on changes to low-frequency components (keeping them "slow") while allowing high-frequency components more flexibility to adapt ("fast") to new class information.

Inter-space SvF: Feature Space Composition

At a coarser grain, the framework maintains two distinct feature spaces: a "Slow-Updated Space" for stability and a "Fast-Updated Space" for plasticity. These spaces are merged using a composition function (such as concatenation or PCA-based fusion) to create a final composite feature space. This duality ensures that the model can leverage both stable base features and specialized new features for classification.

Results and Discussion

MgSvF was evaluated on CIFAR100, CUB200-2011, and MiniImageNet. It significantly outperformed existing state-of-the-art methods. For instance, on the CUB200 benchmark, MgSvF achieved a final accuracy of 54.33% after 10 incremental steps, surpassing the previous best baseline by over 11%. The results demonstrate that the combination of frequency-aware regularization and dual-space composition provides a robust solution to the SvF dilemma, maintaining a lead of 5-8% over competitors throughout all incremental sessions on general object datasets.

Conclusion

By disentangling features into frequency components and maintaining complementary feature spaces, the MgSvF framework effectively balances the competing demands of stability and plasticity in FSCIL. This multi-grained approach sets a new standard for performance in incremental learning scenarios with limited data.

Summary updated

Abstract

As a challenging problem, few-shot class-incremental learning (FSCIL) continually learns a sequence of tasks, confronting the dilemma between slow forgetting of old knowledge and fast adaptation to new knowledge. In this paper, we concentrate on this "slow vs. fast" (SvF) dilemma to determine which knowledge components to be updated in a slow fashion or a fast fashion, and thereby balance old-knowledge preservation and new-knowledge adaptation. We propose a multi-grained SvF learning strategy to cope with the SvF dilemma from two different grains: intra-space (within the same feature space) and inter-space (between two different feature spaces). The proposed strategy designs a novel frequency-aware regularization to boost the intra-space SvF capability, and meanwhile develops a new feature space composition operation to enhance the inter-space SvF learning performance. With the multi-grained SvF learning strategy, our method outperforms the state-of-the-art approaches by a large margin.
BibTeX
@article{zhao2024mgsvf,
  author = {Hanbin Zhao and Yongjian Fu and Mintong Kang and Qi Tian and Fei Wu and Xi Li},
  title = {{MgSvF: Multi-Grained Slow vs. Fast Framework for Few-Shot Class-Incremental Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2021.3133897},
}
Figure