Introduction
Continual learning (CL) aims to train a single model on a sequence of tasks without forgetting previous ones. While many methods tackle catastrophic forgetting, two major hurdles remain for real-world deployment:
- Scalability: Most models either have fixed capacity (leading to interference between tasks) or grow linearly in size, which is unsustainable for hundreds or thousands of tasks.
- Order-Sensitivity: The final performance of a model often depends heavily on the sequence in which tasks were learned. This is a significant fairness issue; for instance, a medical diagnostic AI shouldn't perform worse on a specific disease just because it encountered that disease early in its training life.
The authors identify that "semantic drift"—where shared weights move away from the optimal configuration for earlier tasks—is the primary culprit for both forgetting and order-sensitivity. To address this, they propose Additive Parameter Decomposition (APD), a novel continual learning framework that decomposes network weights into shared and sparse task-specific components.
The Additive Parameter Decomposition (APD) Framework
The core innovation of this paper is APD. Instead of learning a single set of shared weights, the parameters for a given task $t$ ($\theta_t$) are represented as:
$$ \theta_t = \sigma \otimes M_t + \tau_t $$
- $\sigma$ (Task-Shared Parameters): A global weight matrix that captures generic knowledge across all tasks.
- $M_t$ (Task-Adaptive Mask): A learnable attention-like vector that selects which parts of the shared knowledge are relevant to the current task.
- $\tau_t$ (Task-Adaptive Parameters): A sparse, task-specific "residual" that captures idiosyncrasies the shared weights cannot explain.
Key Algorithmic Additions
- Retroactive Parameter Updates: When a new task arrives and the shared $\sigma$ is updated, the model "retroactively" adjusts the $\tau$ values of previous tasks. This ensures that even if the shared knowledge drifts, the task-specific residuals compensate to keep the old solutions intact.
- Hierarchical Knowledge Consolidation: To prevent the number of $\tau$ parameters from exploding, the model periodically clusters similar $\tau$ vectors. This creates "hierarchically shared" parameters, allowing the network capacity to grow logarithmically rather than linearly.
Key Insights and Ideas
- Residual Learning for Tasks: By treating task-specific parameters ($\tau_t$) as residuals, the model only adds what is strictly necessary. If the shared $\sigma$ is already good, $\tau_t$ remains extremely sparse.
- Decoupling Drift from Forgetting: In standard CL, updating shared weights for Task B breaks Task A. In APD, Task A's $\tau_A$ is updated to "counter-steer" against the changes in $\sigma$, effectively neutralizing the drift.
- Selective Forgetting: Because tasks are modularized via $\tau_t$, the model can "forget" a specific task (e.g., for privacy or data obsolescence) simply by deleting its specific $\tau$ and mask, without affecting any other task.
Experimental Results
The researchers evaluated APD-Net on several standard and large-scale benchmarks, including CIFAR-100 (Split and Superclass), Omniglot-rotation (a massive-scale experiment consisting of 100 tasks), and heterogeneous sequences involving entirely different domains (e.g., SVHN $\rightarrow$ CIFAR-10 $\rightarrow$ CIFAR-100).
- Accuracy & Forgetting: APD-Net consistently outperformed state-of-the-art baselines like EWC, PGN, and DEN. On CIFAR-100 Split, it achieved ~60.7% accuracy compared to EWC's 53.7%. As shown in the results, APD maintains high accuracy on the first task even after 100 tasks, effectively eliminating catastrophic forgetting.
- Order-Robustness: The authors measured "Average Order-normalized Performance Disparity" (AOPD). APD-Net showed significantly lower disparity (1.79% on CIFAR-100) compared to baselines (7-8%), meaning its performance is highly stable regardless of the task arrival order.
- Efficiency & Scalability: In the 100-task Omniglot experiment, APD-Net achieved performance nearly equal to training 100 separate models but used 10x fewer parameters than expansion-based competitors like PGN, demonstrating logarithmic memory growth.
Limitations and Future Work
- Training Complexity: The retroactive update step requires iterating through previous task parameters during the training of a new task. While the authors use a "selective update" to speed this up, it still adds computational overhead compared to simple regularization methods.
- Mask Granularity: The current implementation uses neuron/filter-wise masking. Exploring finer-grained (weight-wise) or coarser-grained masking could yield different trade-offs between accuracy and sparsity.
- Task Boundaries: Like most CL research, this paper assumes clear task boundaries (the model knows when a new task starts). Adapting APD to "blurry" or task-free continual learning remains an open challenge.
Summary updated