Abstract
Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.
Introduction
Deep learning models are notoriously bad at multitasking over time. When a model trained on Task A is fine-tuned on Task B, new gradients typically overwrite parameters essential for Task A—a phenomenon known as catastrophic forgetting. Existing solutions generally fall into two categories: regularization methods, which often suffer from capacity limits and performance degradation, and dynamic architectures, which can lead to unsustainable linear growth in model size. This paper proposes a "middle way" that guarantees zero forgetting, encourages knowledge transfer, and maintains model compactness.
The CPG Framework
The authors introduce CPG (Compacting, Picking, and Growing), a three-step iterative loop applied to every new task:
- Compacting (Pruning): After learning a task, the model undergoes gradual pruning to identify and remove weights that do not significantly contribute to performance. These released weights are then available for future tasks.
- Picking (Masking): For a new task, the model uses a learnable binary mask to "pick" specific weights already learned from previous tasks. This enables knowledge reuse (transfer learning) without modifying original weights.
- Growing (Expansion): If the reused weights and available pruned space are insufficient to meet performance goals, the architecture "grows" by adding a small number of new filters or nodes.
Crucially, weights assigned to a task are frozen forever, ensuring that performance on earlier tasks remains unchanged as the sequence progresses.
Experimental Evaluation
The CPG framework was evaluated across three diverse scenarios:
- CIFAR-100: Split into 20 sequential tasks (5 classes each) to test long-term sustainability.
- Fine-grained Classification: A sequence of complex datasets including ImageNet, CUBS, Stanford Cars, Flowers-102, WikiArt, and Sketch.
- Facial Informatics: A real-world pipeline starting with face verification and adding gender, expression, and age classification sequentially.
Key Results
The experimental results highlight several advantages of the CPG approach:
- Zero Forgetting: As shown in the performance comparisons, CPG maintains a flat line for previous task performance, whereas methods like DEN (Dynamic-expansion Net) show significant degradation as more tasks are added.
- Superior Accuracy: On the 20-task CIFAR-100 benchmark, CPG achieved 80.9% average accuracy, significantly outperforming PackNet (67.5%) and individual task training (78.8%).
- Efficiency: CPG is highly parameter-efficient. For 20 tasks, the model grew to only 2.4x its original size, compared to the 20x storage required for independent models.
- Transfer Success: The "Picking" mechanism successfully leveraged accumulated experience, often outperforming training from scratch.
Conclusion and Future Work
CPG provides a robust solution for continual learning by integrating pruning, masking, and conditional expansion. While it currently requires explicit task boundaries and introduces some inference overhead due to binary masks, it offers a sustainable path for lifelong learning. Future work includes exploring structured channel pruning for hardware efficiency and "selective forgetting" to manage model size over infinite timelines.
Summary updated