ScrollNet: Dynamic Weight Importance for Continual Learning

Fei Yang, Kai Wang, Joost van de WeijerWorkshops @ ICCV 2023 · DOI · OpenReview

Abstract

ScrollNet introduces a proactive "scrolling" mechanism for neural networks to address catastrophic forgetting in continual learning. By pre-assigning weight importance and rotating parameter prioritization across sequential tasks, ScrollNet achieves a superior balance between plasticity and stability. This structural modification is orthogonal to existing methods and demonstrates significant performance gains on standard benchmarks like CIFAR100 and TinyImageNet.

Introduction

Deep learning models typically suffer from catastrophic forgetting: as they are trained on a sequence of tasks, they tend to overwrite the weights necessary for previous tasks to accommodate new information. This challenge, known as the "stability-plasticity dilemma," requires models to be stable enough to retain past knowledge while remaining plastic enough to learn new concepts.

While existing Continual Learning (CL) methods like Elastic Weight Consolidation (EWC) or Memory Aware Synapses (MAS) calculate weight importance reactively during or after training, ScrollNet proposes a proactive approach. It pre-defines a structural hierarchy of importance within the network and "scrolls" through this hierarchy as new tasks arrive.

Methodology

ScrollNet is built upon the architecture of slimmable neural networks and operates through three primary steps:

  1. Weight Splitting: The model's parameters are divided into $N$ non-overlapping sets.
  2. Importance Ranking: Sub-networks of increasing size are defined. The smallest sub-network, utilizing the first set of weights, is designated as the most "important" for the current task.
  3. The Scrolling Operation: At the onset of a new task, the network "scrolls." Parameters that were least important for the previous task are rotated to become the foundational, most important weights for the current task.
The scrolling mechanism: weight importance is rotated between tasks to balance plasticity and stability.

This "roller" metaphor (illustrated above) ensures that the primary learning capacity is always focused on "fresh" weights, while previously learned knowledge is shifted to more stable positions. During training, a composite loss function optimizes performance across all sub-network tiers, ensuring the most important weights capture the task's essence while others provide auxiliary support.

Weight rotation across network layers for sequential tasks.

As shown in the network graph, this rotation happens across the network's layers, effectively remapping weights to different functional roles as the task index progresses. Because ScrollNet is a structural modification, it is orthogonal to other CL methods and can be combined with regularization-based or replay-based approaches to further boost performance.

Experiments and Results

The authors evaluated ScrollNet on two standard computer vision benchmarks: CIFAR100 (split into 5, 10, and 20 tasks) and TinyImageNet (200 classes). These were tested in both task-aware and task-agnostic (class-incremental) settings.

Key findings include:

  • Significant Accuracy Gains: Combined with EWC on a 5-task CIFAR100 split, ScrollNet improved task-agnostic accuracy by 9.4% compared to standard EWC.
  • Robustness in Complex Settings: In a 20-task class-incremental setting on CIFAR100, ScrollNet-4 combined with MAS achieved 15.50% accuracy, nearly doubling the performance of the standalone MAS baseline (~9.7%).
  • Granularity Matters: ScrollNet-4 (4-way split) generally outperformed ScrollNet-2, suggesting that finer-grained importance ranking leads to a better stability-plasticity tradeoff.

Discussion and Conclusion

ScrollNet's primary innovation is the use of an "architectural hint"—a structural prior that helps the model organize knowledge efficiently from the start. Its universal compatibility allows it to boost the performance of various baseline CL algorithms consistently across different scales.

However, the method does introduce some training overhead due to the multiple forward passes required for different sub-network tiers. Future research may explore more sophisticated weight-splitting techniques (beyond simple channel splitting) and dynamic "scroll" step sizes based on task correlation to further optimize the learning process.

Summary updated

Abstract

The principle underlying most existing continual learning (CL) methods is to prioritize stability by penalizing changes in parameters crucial to old tasks, while allowing for plasticity in other parameters. The importance of weights for each task can be determined either explicitly through learning a task-specific mask during training (e.g., parameter isolation-based approaches) or implicitly by introducing a regularization term (e.g., regularization-based approaches). However, all these methods assume that the importance of weights for each task is unknown prior to data exposure. In this paper, we propose ScrollNet as a scrolling neural network for continual learning. ScrollNet can be seen as a dynamic network that assigns the ranking of weight importance for each task before data exposure, thus achieving a more favorable stability-plasticity tradeoff during sequential task learning by reassigning this ranking for different tasks. Additionally, we demonstrate that ScrollNet can be combined with various CL methods, including regularization-based and replay-based approaches. Experimental results on CIFAR100 and TinyImagenet datasets show the effectiveness of our proposed method. We release our code at https://github.com/FireFYF/ScrollNet.git.
BibTeX
@inproceedings{yang2023scrollnet,
  author = {Fei Yang and Kai Wang and Joost van de Weijer},
  title = {{ScrollNet: Dynamic Weight Importance for Continual Learning}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00359},
}
Figure