Abstract
ScrollNet introduces a proactive "scrolling" mechanism for neural networks to address catastrophic forgetting in continual learning. By pre-assigning weight importance and rotating parameter prioritization across sequential tasks, ScrollNet achieves a superior balance between plasticity and stability. This structural modification is orthogonal to existing methods and demonstrates significant performance gains on standard benchmarks like CIFAR100 and TinyImageNet.
Introduction
Deep learning models typically suffer from catastrophic forgetting: as they are trained on a sequence of tasks, they tend to overwrite the weights necessary for previous tasks to accommodate new information. This challenge, known as the "stability-plasticity dilemma," requires models to be stable enough to retain past knowledge while remaining plastic enough to learn new concepts.
While existing Continual Learning (CL) methods like Elastic Weight Consolidation (EWC) or Memory Aware Synapses (MAS) calculate weight importance reactively during or after training, ScrollNet proposes a proactive approach. It pre-defines a structural hierarchy of importance within the network and "scrolls" through this hierarchy as new tasks arrive.
Methodology
ScrollNet is built upon the architecture of slimmable neural networks and operates through three primary steps:
- Weight Splitting: The model's parameters are divided into $N$ non-overlapping sets.
- Importance Ranking: Sub-networks of increasing size are defined. The smallest sub-network, utilizing the first set of weights, is designated as the most "important" for the current task.
- The Scrolling Operation: At the onset of a new task, the network "scrolls." Parameters that were least important for the previous task are rotated to become the foundational, most important weights for the current task.
This "roller" metaphor (illustrated above) ensures that the primary learning capacity is always focused on "fresh" weights, while previously learned knowledge is shifted to more stable positions. During training, a composite loss function optimizes performance across all sub-network tiers, ensuring the most important weights capture the task's essence while others provide auxiliary support.
As shown in the network graph, this rotation happens across the network's layers, effectively remapping weights to different functional roles as the task index progresses. Because ScrollNet is a structural modification, it is orthogonal to other CL methods and can be combined with regularization-based or replay-based approaches to further boost performance.
Experiments and Results
The authors evaluated ScrollNet on two standard computer vision benchmarks: CIFAR100 (split into 5, 10, and 20 tasks) and TinyImageNet (200 classes). These were tested in both task-aware and task-agnostic (class-incremental) settings.
Key findings include:
- Significant Accuracy Gains: Combined with EWC on a 5-task CIFAR100 split, ScrollNet improved task-agnostic accuracy by 9.4% compared to standard EWC.
- Robustness in Complex Settings: In a 20-task class-incremental setting on CIFAR100,
ScrollNet-4combined with MAS achieved 15.50% accuracy, nearly doubling the performance of the standalone MAS baseline (~9.7%). - Granularity Matters:
ScrollNet-4(4-way split) generally outperformedScrollNet-2, suggesting that finer-grained importance ranking leads to a better stability-plasticity tradeoff.
Discussion and Conclusion
ScrollNet's primary innovation is the use of an "architectural hint"—a structural prior that helps the model organize knowledge efficiently from the start. Its universal compatibility allows it to boost the performance of various baseline CL algorithms consistently across different scales.
However, the method does introduce some training overhead due to the multiple forward passes required for different sub-network tiers. Future research may explore more sophisticated weight-splitting techniques (beyond simple channel splitting) and dynamic "scroll" step sizes based on task correlation to further optimize the learning process.
Summary updated