Compacting, Picking and Growing for Unforgetting Continual Learning

Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, Chu-Song Chen

TL;DR

The CPG (Compacting, Picking, and Growing) framework solves the "catastrophic forgetting" problem in continual learning by freezing old weights and using learnable masks to "pick" relevant knowledge for new tasks. Unlike previous methods that either forget over time or explode in size, CPG maintains a compact model that can learn an unlimited sequence of tasks with zero performance loss on earlier ones.

What Problem Does This Paper Address?

Deep learning models are notoriously bad at "lifelong learning." When you train a model on Task A and then fine-tune it on Task B, the model usually overwrites the weights needed for Task A, leading to catastrophic forgetting.

Existing solutions fall into two flawed camps:

  1. Regularization/Memory Replay: These try to slow down forgetting but eventually fail as the model's fixed capacity fills up.
  2. Dynamic Architecture: These (like Progressive Networks) add new neurons for every task. While they don't forget, the model size grows linearly with the number of tasks, becoming massive and redundant very quickly.

The authors aim for the "Goldilocks" zone: a model that never forgets, reuses old knowledge efficiently, and stays compact.

Methods

The authors propose a three-step iterative loop called CPG:

The CPG framework illustrating the iterative process of compacting, picking, and growing for sequential tasks.
  1. Compacting: After training a task, the model undergoes "gradual pruning." It identifies and removes redundant weights (setting them to zero) without losing accuracy. These "released" weights become available for future tasks.
  2. Picking: When a new task arrives, the model doesn't just use all previous weights. It trains a learnable binary mask (using a "piggyback" approach) to select only the "critical weights" from previous tasks that are actually useful for the new challenge.
  3. Growing: If the performance goal isn't met using existing weights and the "released" space, the model selectively adds new filters or nodes.

Crucially, once a weight is assigned to a task, it is frozen. This ensures "unforgetting"—the performance on Task 1 will be exactly the same after Task 100 is learned.

Datasets

The framework was evaluated across three diverse scenarios:

  • CIFAR-100: Split into 20 sequential tasks (5 classes each).
  • Fine-grained Classification: A sequence of six distinct datasets: ImageNet, CUBS (birds), Stanford Cars, Flowers, WikiArt, and Sketch.
  • Facial-informatic Tasks: A real-world sequence of face verification, gender classification, expression recognition, and age estimation.

Key Insights and Ideas

  • The "Inertia" of Old Weights: The authors found that using all previous weights for a new task can actually slow down learning (a sort of "knowledge interference"). The "Picking" mechanism acts as a filter, allowing the model to ignore irrelevant old data.
  • Pruning as a Resource Generator: Usually, pruning is just for deployment. Here, it’s a core part of the learning process—it "clears desk space" so the model has room to learn new things without immediately needing to grow the architecture.
  • Sustainability: Because the model removes redundancy at every step, it can handle many more tasks than traditional dynamic networks before the hardware limits are reached.

Key Results

  • Zero Forgetting: Unlike the popular DEN (Dynamic-expansion Net) method, which saw accuracy on Task 1 drop significantly as Task 20 was learned, CPG maintained 100% of its original performance on all previous tasks.
Performance change across sequential tasks, showing CPG maintains 100% accuracy on older tasks while DEN suffers from catastrophic forgetting.
  • Superior Accuracy: On the 20-task CIFAR-100 benchmark, CPG achieved an average accuracy of 80.9%, significantly outperforming standard fine-tuning (78.6%) and the previous state-of-the-art PackNet (67.5%).
  • Efficiency: For the 20 CIFAR tasks, CPG only increased the total number of weights by 1.09x compared to a single-task model. In contrast, training 20 independent models would require 20x the parameters, and Progressive Networks would grow even larger.

Limitations and Future Work

  • Task Boundaries: The current method assumes the model knows exactly when one task ends and another begins (task-based learning). It is not yet "task-free."
  • Weight vs. Channel Pruning: The paper focuses on weight-level pruning, which is great for theoretical compactness but harder to accelerate on standard GPUs compared to "channel pruning" (removing entire neurons). The authors mention exploring channel pruning in future iterations.
  • Selective Forgetting: The authors suggest that in the future, they might implement a way to "selectively forget" truly useless tasks to free up even more space, rather than keeping everything forever.

Summary updated

Abstract

Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.
BibTeX
@inproceedings{hung2019compacting,
  author = {Steven C. Y. Hung and Cheng-Hao Tu and Cheng-En Wu and Chien-Hung Chen and Yi-Ming Chan and Chu-Song Chen},
  title = {{Compacting, Picking and Growing for Unforgetting Continual Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=8xVGWwHIDu},
}
Figure