Neural Weight Search for Scalable Task Incremental Learning

Jian Jiang, Oya CeliktutanWACV 2023

TL;DR

This paper introduces Neural Weight Search (NWS), a novel method for task incremental learning that tackles the challenge of ever-growing model sizes. NWS achieves this by searching for optimal combinations of pre-trained, grouped weights from fixed "pools" to build new task-specific models, drastically reducing memory footprint while maintaining high accuracy.

What Problem Does This Paper Address?

Modern AI systems, especially those deployed in real-world scenarios like robotics, need to learn continuously from new data without forgetting what they've already learned. This is known as incremental learning or continual learning. A specific challenge within this is Task Incremental Learning (TIL), where the system encounters distinct tasks sequentially.

A major hurdle in TIL is catastrophic forgetting, where learning a new task degrades performance on previously learned ones. While some methods try to mitigate this by building separate networks or sub-networks for each new task, this leads to an uncontrollable growth in model size and memory requirements. This is particularly problematic for resource-constrained systems like robots. Existing approaches like Progressive Neural Networks (PNN) or methods using kernel masks (PiggyBack, PackNet, KSM) still struggle with scalability or have limitations in representation capacity. This paper aims to solve the problem of scalable memory growth in TIL while preserving performance.

Methods

The core innovation is Neural Weight Search (NWS). Instead of relying on a fixed backbone or learning masks, NWS takes a "backbone-free" approach. For each layer of a neural network, NWS maintains a fixed "pool" of pre-trained, grouped weights (e.g., convolution kernels). When a new task arrives, NWS searches within these pools to find an optimal combination of these pre-trained weights to construct a new model for that task.

Key aspects of the NWS method:

  • Layer-wise Pools: Each layer has its own pool of indexed, grouped weights. These pools are learned once during a pre-training phase on a large dataset.
  • Weight Search: For a new task, NWS searches for the best combination of weights from these pools to form the new model. This search is guided by a loss function that includes classification accuracy and a similarity loss.
  • Temporary Weights: During training, temporary weights act as a scaffold. These are used to facilitate the search process and are discarded after the model for the new task is built. They are not used for actual input processing.
  • Index-Based Storage: Once the optimal combination of weights for a task is found, the model is stored not by saving all the weights, but by saving the indices of the selected weights from the pools. This is the key to memory reduction.
  • Knowledge Distillation for Pool Construction: To create these pools efficiently, the authors propose a knowledge distillation strategy that distills knowledge from a large-scale dataset into a compact search space with a predefined pool size.

Datasets

  • ImageNet: Used for pre-training the layer-wise weight pools.
  • Split-CIFAR-100: A benchmark for Task Incremental Learning, consisting of 20 tasks, each with 5-way classification on 32x32 images.
  • CUB-to-Sketches: Another TIL benchmark comprising 5 datasets (CUB-200, Cars-196, Flowers-102, WikiArt-195, Sketches-250) with higher resolution images (224x224).

Key Insights and Ideas

  • Decoupling Weight Storage from Model Architecture: NWS decouples the storage of learned knowledge (in the form of weight pools and indices) from the specific architecture of the model for each task. This is a fundamental shift from methods that grow the model itself.
  • Leveraging Pre-trained Weights for Scalability: By reusing pre-trained weights from fixed pools, NWS avoids the need to train new weights from scratch for every task, significantly reducing memory and computational overhead.
  • The Power of Indices: Storing models as indices into pre-defined weight pools is a highly efficient compression strategy, leading to dramatic memory savings.
  • Flexibility Across Architectures: The NWS approach is demonstrated to be compatible with various modern deep neural network architectures (ResNet, MobileNetV2, VGG), suggesting its broad applicability.

Key Results

  • Significant Memory Reduction: NWS achieves substantial memory savings, up to 96% on Split-CIFAR-100 and 95% on CUB-to-Sketches compared to Finetune. It also outperforms other baselines like PackNet and KSM in memory efficiency.
  • State-of-the-Art Accuracy: The method achieves competitive, often state-of-the-art, average accuracy across tasks on both benchmarks. For instance, on Split-CIFAR-100, it outperforms KSM by 1.9% in average accuracy.
  • Scalable Performance: The memory cost grows controllably with the number of tasks, unlike methods that add new parameters for each task.
  • Effective Kernel Utilization: Analysis shows that NWS can effectively utilize kernels from the pools, with higher utilization in later layers, suggesting good specialization.

Limitations and Future Work

  • Pretraining Cost: The initial pre-training of the weight pools on a large dataset like ImageNet can be computationally intensive.
  • Search Space Design: While the paper proposes a method for constructing pools, the optimal design and size of these pools for different tasks and architectures might require further investigation.
  • Task ID Requirement: Like other Task Incremental Learning methods, NWS implicitly requires the task ID during inference to select the correct model (built from specific indices). The paper mentions practical applications where a robot can query the user for the task ID.
  • Future Work: The authors suggest exploring NWS with different grouping strategies for weights (beyond convolution kernels) and investigating its application in more complex scenarios. They also hint at the potential for NWS to be integrated with other continual learning techniques.

Summary updated

BibTeX
@inproceedings{jiang2023neural,
  author = {Jian Jiang and Oya Celiktutan},
  title = {{Neural Weight Search for Scalable Task Incremental Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure