Abstract
Prompt-based continual learning has emerged as a leading paradigm for leveraging pre-trained knowledge in downstream tasks, achieving near-peak performance under supervised pre-training. However, this research identifies a critical "obscured sub-optimality": current strategies falter under more realistic self-supervised pre-training (SSL) conditions. To address this, we introduce HiDe-Prompt (Hierarchical Decomposition Prompt), a framework that decomposes the continual learning objective into three hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. By explicitly optimizing these components through a prompt ensemble and statistical representation modeling, HiDe-Prompt achieves state-of-the-art performance across diverse benchmarks without requiring the storage of raw data.
Introduction
Continual Learning (CL) seeks to enable artificial intelligence models to learn a sequence of tasks without suffering from "catastrophic forgetting." Recent advancements have shifted toward prompt-based methods, which keep a large pre-trained backbone (e.g., a Vision Transformer) frozen and only update a small set of learnable parameters called "prompts." While these methods excel when the backbone is pre-trained using supervised labels, their performance degrades significantly when applied to backbones trained via self-supervised learning (SSL).
This discrepancy is problematic because SSL is the primary method for training massive models on unlabeled, real-world data. Our research reveals that SSL representations are more general and less task-discriminative than supervised ones, making it difficult for standard prompting strategies to incorporate task-specific knowledge or accurately predict task identity at test time.
The Challenge: Obscured Sub-optimality
Existing prompt-based methods like L2P and CODA-Prompt rely on the high discriminative power of supervised pre-training. When switched to SSL backbones (such as iBOT or DINO), these methods struggle with two main issues:
- Representation Overlap: Task-specific knowledge is not effectively separated, leading to interference between tasks.
- Inference Failure: The model fails to correctly identify which task a test sample belongs to, leading to incorrect classification across the global label space.
The HiDe-Prompt Framework
To overcome these challenges, we propose HiDe-Prompt, which breaks the complex CL objective into three manageable, hierarchical sub-problems:
1. Within-Task Prediction (WTP)
WTP focuses on distinguishing classes within the current task. HiDe-Prompt utilizes a Prompt Ensemble strategy, where new prompts are initialized from previous ones to maintain continuity. To prevent feature overlap, we implement a Contrastive Regularization strategy that ensures the representations of the current task remain distinct from those of previous tasks.
2. Task-Identity Inference (TII)
TII involves determining which task a specific sample belongs to at test time. Instead of storing old images—which poses privacy and storage risks—HiDe-Prompt saves the mean and covariance (statistics) of the features for each class. These statistics are used to generate "pseudo-features" that train a lightweight auxiliary classifier to identify the task origin of an input.
3. Task-Adaptive Prediction (TAP)
TAP is the final step of making a classification across all seen classes. By combining the task-specific insights from WTP and the task-identity from TII, the model can make a final, informed prediction. The use of statistical modeling allows the model to "rehearse" in the feature space, effectively simulating old data without needing the original images.
Experimental Results
We evaluated HiDe-Prompt against several benchmarks, including Split CIFAR-100, Split ImageNet-R, and the 5-Datasets mix. We tested across various pre-training paradigms, including supervised (Sup-21K) and SSL methods (iBOT, DINO, and MoCo v3).
Performance Gains
HiDe-Prompt consistently outperformed existing rehearsal-free methods, with the most dramatic improvements seen on SSL backbones:
- Split CIFAR-100 (iBOT): Achieved 93.48% accuracy, representing a 14.37% lead over CODA-Prompt.
- Split ImageNet-R: Maintained a lead of up to 9.61% over competitors.
- Forgetting: The Final Forgetting Measure (FFM) remained near zero in several tests, indicating nearly perfect retention of prior knowledge.
Efficiency
Beyond accuracy, HiDe-Prompt is more resource-efficient. By separating prompt construction from task inference, it requires significantly less GPU memory (e.g., ~16GB vs. ~25GB for CODA-Prompt on CIFAR-100), making it more viable for real-world deployment.
Conclusion and Future Work
HiDe-Prompt addresses the hidden sub-optimality of prompt-based CL by rethinking how task-specific knowledge is organized and retrieved. By decomposing the problem into hierarchical components and utilizing statistical feature rehearsal, it bridges the gap between supervised and self-supervised pre-training performance.
Future research will focus on adapting this framework for non-Transformer architectures and exploring "slow-fast" systems where the backbone can be updated gradually without losing its foundational knowledge. As it stands, HiDe-Prompt provides a robust, efficient, and privacy-preserving path forward for continual learning in the era of massive self-supervised models.
Summary updated