How do humans learn a new language without forgetting how to speak their first? In the world of Artificial Intelligence, this is known as Continual Learning (CL), and it is notoriously difficult. Most neural networks suffer from "catastrophic forgetting"—when they learn a new task, the new information effectively overwrites the old.
A new paper from researchers at East China Normal University and Tencent YouTu Lab proposes a clever solution: stop treating different types of learning as a single process and start alternating between them. Their "Alternate Learner" framework disentangles self-supervised learning (SSL) and supervised learning (SL) to combat forgetting.
The Stability-Plasticity Dilemma
The core challenge in Continual Learning is the tension between plasticity (the ability to learn new things) and stability (the ability to remember old things). Standard Supervised Learning (SL) is excellent for plasticity but poor for stability; it focuses on category labels, which are quickly overwritten as new tasks arrive. Recent attempts have turned to Self-Supervised Learning (SSL) to provide stability by teaching the model to recognize individual instances (instance-level discrimination) without labels.
However, the authors point out a fundamental bottleneck: current SSL objectives involve a trade-off between being invariant to image distortions (like cropping or color jittering) and preserving the original sample's information. This conflict prevents models from reaching their full potential in maintaining a stable feature space over time.
Disentangling the SSL Objective
Instead of a single, conflicted loss function, the researchers reformulate SSL into two distinct goals:
- Maximizing Sample Information: Using a loss based on KL-divergence ($\mathcal{L}^s_{SSL}$) to ensure the model's embeddings retain as much information about the original image as possible.
- Promoting Invariance: Using a loss based on Jensen-Shannon divergence ($\mathcal{L}^c_{SSL}$) to ensure the model produces compact, similar representations for different augmented views of the same image.
By disentangling these objectives, the model can produce much more compact embeddings (128 dimensions) compared to previous methods that required high-dimensional outputs (2048 dimensions) to achieve similar results.
The Alternate Learner Paradigm
The training process is structured into two loops to effectively manage the integration of new and old knowledge:
- The Inner Loop: For each new task, the model alternates between SSL and SL. SSL acts as a "pre-training" step that maintains stability and instance-level features, while SL uses category labels to adapt the model to the specific task.
- The Outer Loop: Inspired by meta-learning algorithms like Reptile, the authors use a momentum-based update to aggregate the parameters learned in the inner loop. This ensures the "global" model moves in a direction that is beneficial for all tasks, not just the most recent one.
Key Results and Performance
The proposed method consistently outperformed existing state-of-the-art (SOTA) models across major computer vision benchmarks, including CIFAR-100 and ImageNet-1K. On a 10-task split of CIFAR-100, it achieved 79.16% average accuracy, surpassing the previous best (DyTox+) by over 2.4%. On the large-scale ImageNet-1K benchmark, it outperformed FOSTER and DER by roughly 4.4% to 4.9%.
Despite the double-loop structure, the computational cost only grows slightly (about 1.04x per task) and remains manageable, with an average cost of 1.42x compared to a standard baseline. This demonstrates that alternating between instance-level and category-level supervision is a powerful and efficient strategy for building stable, lifelong learners.
Summary updated