Few-Shot Lifelong Learning

Pratik Mazumder, Pravendra Singh, Piyush Rai

Abstract

Many real-world classification problems often have classes with very few labeled training samples. Moreover, all possible classes may not be initially available for training, and may be given incrementally. Deep learning models need to deal with this two-fold problem in order to perform well in real-life situations. In this paper, we propose a novel Few-Shot Lifelong Learning (FSLL) method that enables deep learning models to perform lifelong/continual learning on few-shot data. Our method selects very few parameters from the model for training every new set of classes instead of training the full model. This helps in preventing overfitting. We choose the few parameters from the model in such a way that only the currently unimportant parameters get selected. By keeping the important parameters in the model intact, our approach minimizes catastrophic forgetting. Furthermore, we minimize the cosine similarity between the new and the old class prototypes in order to maximize their separation, thereby improving the classification performance. We also show that integrating our method with self-supervision improves the model performance significantly. We experimentally show that our method significantly outperforms existing methods on the miniImageNet, CIFAR-100, and CUB-200 datasets. Specifically, we outperform the state-of-the-art method by an absolute margin of 19.27% for the CUB dataset.

Introduction

Standard deep learning models are notoriously "brittle" when faced with real-world constraints. They typically suffer from two major flaws: data hunger, where they require thousands of examples to avoid overfitting, and catastrophic forgetting, where learning new tasks overwrites the weights necessary for previous ones. This paper addresses these issues through a framework for Few-Shot Class-Incremental Learning (FSCIL), enabling models to learn new concepts from minimal data without losing prior knowledge.

Methodology: Selective Parameter Adaptation

The core philosophy of FSLL is minimal intervention. Rather than retraining the entire network for new tasks, the authors propose a multi-faceted strategy to balance plasticity and stability.

Parameter Selection and Regularization

The researchers argue that weight magnitude is a reliable proxy for knowledge; weights with high absolute values are "important" for existing tasks. FSLL identifies the bottom ~10% of weights (those with the lowest magnitude) and designates them as "session trainable parameters." The remaining 90% are frozen to preserve old memories. To ensure these updates remain stable, an $L_1$ regularization penalty is applied to prevent the parameters from drifting too far from their original state.

Conceptual overview of the FSLL framework showing selective parameter updates across incremental sessions and the resulting separation of class prototypes in feature space.

Feature Space Optimization

To improve classification, FSLL utilizes a triplet loss to cluster images of the same new class while pushing different classes apart. Crucially, a "cosine similarity loss" is employed to explicitly push the prototypes (average feature representations) of new classes away from old ones. Furthermore, the model is pre-trained with a self-supervised auxiliary task—such as rotation prediction—to learn more robust, general-purpose features before the incremental stages begin.

Experimental Results

The FSLL method was evaluated on three standard benchmarks: miniImageNet, CIFAR-100, and the fine-grained CUB-200 dataset.

Performance Benchmarks

FSLL demonstrated a significant performance gap over previous state-of-the-art methods like TOPIC. On the challenging CUB-200 dataset, which requires distinguishing between very similar bird species, FSLL achieved an absolute improvement of 19.27%. Similar trends were observed on miniImageNet and CIFAR-100, with improvements of approximately 15.07% and 9.09%, respectively.

Detailed performance comparison on the CUB-200 dataset across 11 learning sessions, highlighting the significant relative improvements of the FSLL variants over existing methods.

Mitigation of Forgetting

Impressively, the model maintained high accuracy on the "base" classes even after 10 subsequent rounds of learning new classes. This proves that by restricting the model's capacity to change and focusing only on "unimportant" parameters, catastrophic forgetting can be successfully mitigated.

Conclusion

FSLL provides a robust solution for lifelong learning in data-constrained environments. By leveraging weight magnitude for parameter selection and enforcing prototype separation, the method achieves state-of-the-art results on complex fine-grained datasets. Future work may involve dynamic parameter selection to further optimize the balance between learning and retention.

Summary updated

Abstract

Many real-world classification problems often have classes with very few labeled training samples. Moreover, all possible classes may not be initially available for training, and may be given incrementally. Deep learning models need to deal with this two-fold problem in order to perform well in real-life situations. In this paper, we propose a novel Few-Shot Lifelong Learning (FSLL) method that enables deep learning models to perform lifelong/continual learning on few-shot data. Our method selects very few parameters from the model for training every new set of classes instead of training the full model. This helps in preventing overfitting. We choose the few parameters from the model in such a way that only the currently unimportant parameters get selected. By keeping the important parameters in the model intact, our approach minimizes catastrophic forgetting. Furthermore, we minimize the cosine similarity between the new and the old class prototypes in order to maximize their separation, thereby improving the classification performance. We also show that integrating our method with self-supervision improves the model performance significantly. We experimentally show that our method significantly outperforms existing methods on the miniImageNet, CIFAR-100, and CUB-200 datasets. Specifically, we outperform the state-of-the-art method by an absolute margin of 19.27% for the CUB dataset.
BibTeX
@inproceedings{mazumder2021few,
  author = {Pratik Mazumder and Pravendra Singh and Piyush Rai},
  title = {{Few-Shot Lifelong Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16334},
}
Figure