Abstract
While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, some issues remain to be tackled to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.
Introduction
Continual learning aims to enable artificial intelligence to learn a sequence of tasks without suffering from "catastrophic forgetting," where new information overwrites previously learned knowledge. However, existing approaches often struggle with scalability—either hitting capacity limits or growing excessively in size—and order-sensitivity, where the final accuracy depends heavily on the sequence of tasks. This order-sensitivity is particularly problematic in sensitive domains like healthcare, where model fairness is paramount.
To address these challenges, we introduce APD-Net, a framework utilizing Additive Parameter Decomposition (APD). APD-Net decomposes neural network weights into shared and task-specific components, allowing for efficient scaling and robust performance regardless of task order.
Methodology: Additive Parameter Decomposition
The core of our approach is the decomposition of parameters $\theta_t$ for any task $t$ into two distinct parts:
- Task-Shared Parameters ($\sigma$): Global weights capturing generic knowledge across the entire task sequence.
- Task-Adaptive Parameters ($\tau_t$): Sparse, task-specific weights that capture unique features not explained by the shared weights.
To maintain performance across tasks, we employ Retroactive Parameter Updates. When shared weights $\sigma$ are updated for a new task, the model adjusts the task-adaptive weights $\tau$ of previous tasks. This ensures that historical knowledge remains compatible with the evolving shared representation. Furthermore, Hierarchical Knowledge Consolidation clusters similar task-adaptive parameters, allowing tasks to share "local" knowledge and enabling logarithmic rather than linear capacity growth.
Experimental Evaluation
We evaluated APD-Net on several benchmarks, including Split CIFAR-100, Omniglot-rotation (100 tasks), and heterogeneous sequences (SVHN, CIFAR-10/100).
Key Results
- Accuracy and Forgetting: APD-Net consistently outperformed state-of-the-art methods like EWC and PGN. As shown in Figure (a) above, while EWC suffers significant performance drops on the first task as more tasks are added, APD-Net maintains near-constant accuracy.
- Scalability: In the 100-task Omniglot experiment, APD-Net demonstrated superior efficiency. Figure (b) illustrates that APD-Net's capacity increase is significantly lower than that of PGN while maintaining higher performance.
- Order Robustness: APD-Net showed minimal "Performance Disparity" (1.79% on CIFAR-100) compared to baselines like EWC (7.06%). Figure (c) highlights how APD-Net's performance on the 5th task remains stable across different task orderings (Order A-E).
Conclusion
APD-Net provides a scalable and order-robust solution for continual learning. By treating task-specific parameters as sparse residuals and implementing retroactive updates, the model achieves a "bidirectional" knowledge flow that preserves the past while learning for the future. Future work includes extending this to "task-free" scenarios where task boundaries are not explicitly provided and exploring selective forgetting for privacy-preserving AI.
Summary updated