Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

Jaehong Yoon, Saehoon Kim, Eunho Yang, Sung Ju Hwang

Introduction

Continual learning (CL) aims to train a single model on a sequence of tasks without forgetting previous ones. While many methods tackle catastrophic forgetting, two major hurdles remain for real-world deployment:

  1. Scalability: Most models either have fixed capacity (leading to interference between tasks) or grow linearly in size, which is unsustainable for hundreds or thousands of tasks.
  2. Order-Sensitivity: The final performance of a model often depends heavily on the sequence in which tasks were learned. This is a significant fairness issue; for instance, a medical diagnostic AI shouldn't perform worse on a specific disease just because it encountered that disease early in its training life.

The authors identify that "semantic drift"—where shared weights move away from the optimal configuration for earlier tasks—is the primary culprit for both forgetting and order-sensitivity. To address this, they propose Additive Parameter Decomposition (APD), a novel continual learning framework that decomposes network weights into shared and sparse task-specific components.

The Additive Parameter Decomposition (APD) Framework

The core innovation of this paper is APD. Instead of learning a single set of shared weights, the parameters for a given task $t$ ($\theta_t$) are represented as:

$$ \theta_t = \sigma \otimes M_t + \tau_t $$

  • $\sigma$ (Task-Shared Parameters): A global weight matrix that captures generic knowledge across all tasks.
  • $M_t$ (Task-Adaptive Mask): A learnable attention-like vector that selects which parts of the shared knowledge are relevant to the current task.
  • $\tau_t$ (Task-Adaptive Parameters): A sparse, task-specific "residual" that captures idiosyncrasies the shared weights cannot explain.
Diagram illustrating Additive Parameter Decomposition and Hierarchical Knowledge Consolidation.

Key Algorithmic Additions

  • Retroactive Parameter Updates: When a new task arrives and the shared $\sigma$ is updated, the model "retroactively" adjusts the $\tau$ values of previous tasks. This ensures that even if the shared knowledge drifts, the task-specific residuals compensate to keep the old solutions intact.
  • Hierarchical Knowledge Consolidation: To prevent the number of $\tau$ parameters from exploding, the model periodically clusters similar $\tau$ vectors. This creates "hierarchically shared" parameters, allowing the network capacity to grow logarithmically rather than linearly.

Key Insights and Ideas

  • Residual Learning for Tasks: By treating task-specific parameters ($\tau_t$) as residuals, the model only adds what is strictly necessary. If the shared $\sigma$ is already good, $\tau_t$ remains extremely sparse.
  • Decoupling Drift from Forgetting: In standard CL, updating shared weights for Task B breaks Task A. In APD, Task A's $\tau_A$ is updated to "counter-steer" against the changes in $\sigma$, effectively neutralizing the drift.
  • Selective Forgetting: Because tasks are modularized via $\tau_t$, the model can "forget" a specific task (e.g., for privacy or data obsolescence) simply by deleting its specific $\tau$ and mask, without affecting any other task.

Experimental Results

The researchers evaluated APD-Net on several standard and large-scale benchmarks, including CIFAR-100 (Split and Superclass), Omniglot-rotation (a massive-scale experiment consisting of 100 tasks), and heterogeneous sequences involving entirely different domains (e.g., SVHN $\rightarrow$ CIFAR-10 $\rightarrow$ CIFAR-100).

Graphs demonstrating APD's superiority in preventing catastrophic forgetting, maintaining scalability, and ensuring order-robustness compared to baselines.
  • Accuracy & Forgetting: APD-Net consistently outperformed state-of-the-art baselines like EWC, PGN, and DEN. On CIFAR-100 Split, it achieved ~60.7% accuracy compared to EWC's 53.7%. As shown in the results, APD maintains high accuracy on the first task even after 100 tasks, effectively eliminating catastrophic forgetting.
  • Order-Robustness: The authors measured "Average Order-normalized Performance Disparity" (AOPD). APD-Net showed significantly lower disparity (1.79% on CIFAR-100) compared to baselines (7-8%), meaning its performance is highly stable regardless of the task arrival order.
  • Efficiency & Scalability: In the 100-task Omniglot experiment, APD-Net achieved performance nearly equal to training 100 separate models but used 10x fewer parameters than expansion-based competitors like PGN, demonstrating logarithmic memory growth.

Limitations and Future Work

  • Training Complexity: The retroactive update step requires iterating through previous task parameters during the training of a new task. While the authors use a "selective update" to speed this up, it still adds computational overhead compared to simple regularization methods.
  • Mask Granularity: The current implementation uses neuron/filter-wise masking. Exploring finer-grained (weight-wise) or coarser-grained masking could yield different trade-offs between accuracy and sparsity.
  • Task Boundaries: Like most CL research, this paper assumes clear task boundaries (the model knows when a new task starts). Adapting APD to "blurry" or task-free continual learning remains an open challenge.

Summary updated

Abstract

While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, there are issues that remain to be tackled in order to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.
BibTeX
@inproceedings{yoon2020scalable,
  author = {Jaehong Yoon and Saehoon Kim and Eunho Yang and Sung Ju Hwang},
  title = {{Scalable and Order-robust Continual Learning with Additive Parameter Decomposition}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Ds3bHC0Y2A},
}
Figure