Continual Learning with Filter Atom Swapping

Zichen Miao, Ze Wang, Wei Chen, Qiang QiuICLR 2022 · OpenReview

Introduction

Deep neural networks are notoriously bad at "continual learning"—the ability to learn new tasks sequentially without forgetting old ones. This phenomenon, known as catastrophic forgetting, occurs when a model is updated for a new task, overwriting the weights that were optimal for previous tasks.

Current solutions typically fall into two camps:

  1. Replay-based: Storing a subset of old images or gradients to "remind" the model of the past. This is memory-intensive and scales poorly to hundreds of tasks.
  2. Expansion-based: Growing the network for every new task. This prevents forgetting but leads to a "parameter explosion" where the model becomes too large to deploy.

To address this, researchers have developed a Filter Atom Swapping technique that virtually eliminates catastrophic forgetting. By decomposing convolutional filters into shared coefficients and task-specific "atoms," the model can perfectly reconstruct past versions of itself with a memory footprint orders of magnitude smaller than traditional methods.

The Core Method: Filter Atom Decomposition

The core innovation lies in rethinking the convolutional filter. In a standard CNN, a filter $W$ is a fixed block of parameters. The authors propose decomposing this filter into two distinct parts:

  • Task-Shared Coefficients ($\alpha$): These handle "channel mixing." They are learned during the very first task and then permanently frozen.
  • Task-Specific Atoms ($D$): These are lightweight spatial filters (the "atoms").

The relationship is expressed mathematically as $W = \alpha \times D$.

Filter Atom Swapping Pipeline

The "Swapping" Mechanism

To learn a new task, the model keeps the shared coefficients ($\alpha$) fixed and only trains a new set of filter atoms ($D^t$). To "remember" Task 1 while performing Task 10, the model simply swaps the current atoms for the Task 1 atoms stored in a small "atom memory." Because the coefficients are shared, the model perfectly reconstructs the exact weights it had for any previous task, providing a mathematical guarantee against forgetting.

Advanced Ensembling for Better Performance

Beyond basic swapping, the framework enables flexible model ensembling to boost performance:

  1. Inter-task Ensemble: When facing a new task, the model uses Grassmann distance (a measure of distance between mathematical subspaces) to find the most relevant past tasks from its atom memory. It then combines their knowledge to help learn the current task.
  2. Intra-task Ensemble: For "Class-Incremental" settings where the task ID is unknown at inference, the model maintains multiple groups of atoms for a single task. This creates "virtual members" that help the model better predict which task ID a piece of data belongs to.
Inter-Task Ensemble via Subspace Distance

Key Insights and Results

The authors evaluated their method on standard continual learning benchmarks, including 5-Split MNIST, CIFAR-100, miniImageNet, and ImageNet-Subset. The results demonstrate significant improvements over state-of-the-art (SOTA) methods like iCaRL, LUCIR, and ACL.

  • Extreme Memory Efficiency: Because atoms consist of just a few hundred parameters per layer, the "atom memory" is tiny. On the 20-Split CIFAR-100 test, the method required only 0.86 MB of memory, whereas traditional memory-based methods often require hundreds of megabytes to store image exemplars.
  • High Accuracy: On the same 20-Split CIFAR-100 benchmark, the proposed method achieved 80.75% accuracy, outperforming the previous SOTA (ACL) at 78.08%.
  • Scalability: The parameter memory growth stays nearly flat over time, making it highly viable for long-term "life-long" learning without parameter explosion.
  • Zero-Forgetting Guarantee: Unlike replay methods that merely approximate the past, this method archives it. The model's performance on early tasks will never degrade.

Limitations and Future Work

While highly effective, the method has a few limitations. First is the First Task Bias: because the shared coefficients ($\alpha$) are learned on the first task, if that task is not representative of the general data distribution, the mixing logic might be sub-optimal for future tasks. Second, regarding Ensemble Selection, more ensemble members aren't always better; irrelevant past knowledge can sometimes act as noise.

Future work will explore how this decomposition could be applied beyond CNNs (such as to Transformer architectures) and investigate dynamic ways to update the shared coefficients without triggering catastrophic forgetting.

Summary updated

Abstract

Continual learning has been widely studied in recent years to resolve the catastrophic forgetting of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning with filter atom swapping. In other words, we learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. By maintaining a small footprint memory of filter atoms, we can easily archive models for past tasks to avoid forgetting. The effectiveness of this simple scheme for continual learning is illustrated both empirically and theoretically. The proposed atom swapping framework further enables flexible and efficient model ensemble with members selected within a task or across tasks to improve the performance in different continual learning settings. Being validated on multiple benchmark datasets with different convolutional network structures, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.
BibTeX
@inproceedings{miao2022continual,
  author = {Zichen Miao and Ze Wang and Wei Chen and Qiang Qiu},
  title = {{Continual Learning with Filter Atom Swapping}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=qNkppBBmnL},
}
Figure