NTK-Guided Few-Shot Class Incremental Learning

Jingren Liu, Zhong Ji, Yanwei Pang, Yunlong Yu

Abstract

Few-Shot Class Incremental Learning (FSCIL) requires models to continuously learn new classes from limited data without forgetting previously acquired knowledge. This paper introduces a novel framework, NTK-FSCIL, which redefines the "anti-amnesia" challenge as a problem of mathematical generalization. By leveraging Neural Tangent Kernel (NTK) theory, the authors propose a method to ensure global network convergence and minimize generalization loss. The approach achieves state-of-the-art performance across multiple benchmarks, including a 9.3% end-session accuracy gain on CIFAR100.

Introduction

In the field of machine learning, Few-Shot Class Incremental Learning (FSCIL) represents a significant hurdle: the stability-plasticity dilemma. Models must remain plastic enough to integrate new classes from sparse examples while remaining stable enough to preserve the "base" knowledge. Traditional approaches often rely on knowledge distillation or prototype preservation to mitigate catastrophic forgetting. However, this research argues that the root of the problem lies in generalization. If a model generalizes perfectly according to theoretical bounds, it should naturally resist the parameter shifts that cause amnesia when encountering novel classes.

Methodology: The NTK-FSCIL Framework

The authors utilize the Neural Tangent Kernel (NTK)—a mathematical tool describing the behavior of deep neural networks during training as their width approaches infinity—to guide optimization. The NTK-FSCIL framework is built upon three primary pillars designed to align the model's training with theoretical generalization bounds.

1. Global NTK Convergence via Meta-Learning

The framework theoretically aligns meta-learning optimization with NTK convergence. The authors introduce a customized meta-learning strategy that employs a mixup mechanism and an unsupervised "adaptability loss." This strategy simulates incremental learning sessions during the initial training phase, preparing the model for the specific dynamics of few-shot updates.

2. Self-Supervised Initialization

Departing from standard supervised pre-training, the researchers utilize SparK, a generative masked image modeling technique. Their findings suggest that generative pre-training establishes a more stable eigenvalue distribution within the NTK. This mathematical state provides a superior foundation for future learning compared to traditional supervised or contrastive methods.

3. Dual NTK Regularization

To maintain robust NTK properties throughout the learning process, the authors apply two forms of regularization:

  • Convolutional Spectral Regularization: This uses Singular Value Decomposition (SVD) to ensure a slower decay of eigenvalues within convolutional layers, preserving feature diversity.
  • Linear NTK Regularization: This directly constrains the NTK matrix of the final linear layers, minimizing fluctuations between training batches and stabilizing the decision boundaries.

Experimental Results

The NTK-FSCIL method was evaluated on four standard benchmarks: CIFAR100, miniImageNet, CUB200-2011, and ImageNet100. The results demonstrate a significant leap in performance over existing state-of-the-art (SOTA) methods:

  • CIFAR100: The model achieved an end-session accuracy of 61.3%, representing a 9.3% improvement over the previous SOTA (NC-FSCIL).
  • miniImageNet: End-session accuracy improved by 3.5% over the nearest competitor.
  • CUB200-2011: The method showed superior performance even when using standard, non-widened backbones, validating the efficacy of the proposed loss functions.
  • Stability: The "Performance Drop" (PD)—the accuracy gap between the initial and final sessions—was significantly reduced, indicating a high degree of resistance to amnesia.

Key Insights

The Role of Network Width

The research confirms a fundamental NTK principle: increasing a network's width facilitates convergence to a stable matrix, which in turn bolsters generalization. Experiments showed that wider ResNet architectures consistently outperformed narrower versions in incremental tasks.

Generative vs. Contrastive Pre-training

An interesting discovery was that generative self-supervised pre-training (like SparK) outperformed contrastive methods (such as SimCLR or MoCo). Generative tasks appear to capture a blend of local and global features that are more "future-proof" for identifying new categories in later sessions.

Generalization and Spectral Properties

The authors demonstrate that the performance drop typically observed in FSCIL is mathematically linked to the spectral properties of the NTK. By "tightening" the eigenvalue distribution, they can theoretically and empirically reduce generalization loss.

Limitations and Future Directions

Despite its success, NTK-FSCIL faces certain challenges. Computing the full NTK for deep convolutional networks is memory-intensive, often requiring hundreds of gigabytes of GPU RAM. While the authors used SVD and focused on linear layers to bypass this, full-scale NTK optimization remains computationally expensive.

Furthermore, while the theory was primarily developed for ConvNets, preliminary results suggest that Transformers (e.g., Swin or ViT) also benefit from these principles. Future work will likely explore the NTK of Transformers in incremental contexts and attempt to scale these principles to large-scale incremental learning involving thousands of classes.

Summary updated

Abstract

The proliferation of Few-Shot Class Incremental Learning (FSCIL) methodologies has highlighted the critical challenge of maintaining robust anti-amnesia capabilities in FSCIL learners. In this paper, we present a novel conceptualization of anti-amnesia in terms of mathematical generalization, leveraging the Neural Tangent Kernel (NTK) perspective. Our method focuses on two key aspects: ensuring optimal NTK convergence and minimizing NTK-related generalization loss, which serve as the theoretical foundation for cross-task generalization. To achieve global NTK convergence, we introduce a principled meta-learning mechanism that guides optimization within an expanded network architecture. Concurrently, to reduce the NTK-related generalization loss, we systematically optimize its constituent factors. Specifically, we initiate self-supervised pre-training on the base session to enhance NTK-related generalization potential. These self-supervised weights are then carefully refined through curricular alignment, followed by the application of dual NTK regularization tailored specifically for both convolutional and linear layers. Through the combined effects of these measures, our network acquires robust NTK properties, ensuring optimal convergence and stability of the NTK matrix and minimizing the NTK-related generalization loss, significantly enhancing its theoretical generalization. On popular FSCIL benchmark datasets, our NTK-FSCIL surpasses contemporary state-of-the-art approaches, elevating end-session accuracy by 2.9\% to 9.3\%.
BibTeX
@article{liu2024ntk,
  author = {Jingren Liu and Zhong Ji and Yanwei Pang and Yunlong Yu},
  title = {{NTK-Guided Few-Shot Class Incremental Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2024.3478854},
}
Figure