Introduction
Online Continual Learning (CL) presents a significant challenge for deep learning models: the requirement to learn from a non-stationary data stream where each sample is observed only once. Historically, research in this field has focused primarily on stability—the prevention of "catastrophic forgetting" of previously acquired knowledge. However, this paper argues that focusing solely on stability is insufficient. The authors identify plasticity—the model's inherent capacity to effectively acquire new information—as a critical but often neglected bottleneck in online CL performance.
The Plasticity Challenge
A fundamental performance gap exists between offline learning (where data is seen multiple times) and online learning. Even when utilizing memory replay buffers to store past samples, online models struggle to match the learning accuracy of their offline counterparts. To quantify this, the authors derive a relationship for performance:
$$Average Accuracy \approx Plasticity \times (1 - Forgetting)$$
This formula demonstrates that high final accuracy is unattainable without addressing the "plasticity gap." As illustrated in the comparative analysis below, online models exhibit significantly lower learning accuracy compared to offline baselines, regardless of the number of memory iterations.
Methodology: Collaborative Continual Learning with Distillation Chain (CCL-DC)
To bridge this gap, the authors propose Collaborative Continual Learning with Distillation Chain (CCL-DC). This framework introduces two primary innovations:
1. Collaborative Continual Learning (CCL)
Instead of optimizing a single model, CCL employs two peer models ($\theta^1$ and $ heta^2$) that learn from the data stream simultaneously. These models act as peer teachers, distilling knowledge to one another. This parallel optimization provides the learners with greater flexibility and prevents the optimization process from getting stuck in poor local minima early in the stream.
2. Distillation Chain (DC)
The authors discovered that overconfidence in online CL leads to poor generalization. To combat this, they implemented a "hard-to-easy" optimization strategy:
- Augmentation Chain: Input images undergo a sequence of increasingly intense data augmentations, creating a spectrum of "hard" to "easy" samples.
- Distillation Chain: The models perform distillation from less confident predictions (generated from hard samples) to more confident predictions (generated from easier samples). This acts as a form of learned entropy regularization, suppressing overconfidence and encouraging the model to learn more robust, generalizable features.
Experimental Results
The CCL-DC framework was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-100. The method was integrated into six state-of-the-art CL baselines, including Experience Replay (ER) and Dark Experience Replay (DER++).
Key findings include:
- Significant Performance Gains: On Tiny-ImageNet with a memory buffer of 2000, CCL-DC improved the baseline ER accuracy from 10.82% to 16.56%—a 53% relative improvement.
- Enhanced Plasticity: Across all datasets, the "Learning Accuracy" metric saw dramatic increases, often by 10-20 percentage points, confirming that the method successfully improves the model's ability to learn new tasks.
- Faster Convergence: Loss curve analysis indicates that CCL-DC allows models to descend deeper into the loss landscape more rapidly than standard methods.
- Robust Feature Learning: GradCAM visualizations revealed that CCL-DC helps models focus on relevant objects rather than background "shortcuts," leading to more reliable classification.
Conclusion
By shifting the focus from mere stability to active plasticity improvement, CCL-DC provides a powerful new direction for Online Continual Learning. The combination of peer-teaching and entropy-regulating distillation chains effectively bridges the plasticity gap, allowing online models to approach the performance levels of offline learning systems.
Summary updated