Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality

Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, Jun Zhu

Abstract

Prompt-based continual learning has emerged as a leading paradigm for leveraging pre-trained knowledge in downstream tasks, achieving near-peak performance under supervised pre-training. However, this research identifies a critical "obscured sub-optimality": current strategies falter under more realistic self-supervised pre-training (SSL) conditions. To address this, we introduce HiDe-Prompt (Hierarchical Decomposition Prompt), a framework that decomposes the continual learning objective into three hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. By explicitly optimizing these components through a prompt ensemble and statistical representation modeling, HiDe-Prompt achieves state-of-the-art performance across diverse benchmarks without requiring the storage of raw data.

Introduction

Continual Learning (CL) seeks to enable artificial intelligence models to learn a sequence of tasks without suffering from "catastrophic forgetting." Recent advancements have shifted toward prompt-based methods, which keep a large pre-trained backbone (e.g., a Vision Transformer) frozen and only update a small set of learnable parameters called "prompts." While these methods excel when the backbone is pre-trained using supervised labels, their performance degrades significantly when applied to backbones trained via self-supervised learning (SSL).

This discrepancy is problematic because SSL is the primary method for training massive models on unlabeled, real-world data. Our research reveals that SSL representations are more general and less task-discriminative than supervised ones, making it difficult for standard prompting strategies to incorporate task-specific knowledge or accurately predict task identity at test time.

The Challenge: Obscured Sub-optimality

Existing prompt-based methods like L2P and CODA-Prompt rely on the high discriminative power of supervised pre-training. When switched to SSL backbones (such as iBOT or DINO), these methods struggle with two main issues:

  1. Representation Overlap: Task-specific knowledge is not effectively separated, leading to interference between tasks.
  2. Inference Failure: The model fails to correctly identify which task a test sample belongs to, leading to incorrect classification across the global label space.

The HiDe-Prompt Framework

To overcome these challenges, we propose HiDe-Prompt, which breaks the complex CL objective into three manageable, hierarchical sub-problems:

1. Within-Task Prediction (WTP)

WTP focuses on distinguishing classes within the current task. HiDe-Prompt utilizes a Prompt Ensemble strategy, where new prompts are initialized from previous ones to maintain continuity. To prevent feature overlap, we implement a Contrastive Regularization strategy that ensures the representations of the current task remain distinct from those of previous tasks.

2. Task-Identity Inference (TII)

TII involves determining which task a specific sample belongs to at test time. Instead of storing old images—which poses privacy and storage risks—HiDe-Prompt saves the mean and covariance (statistics) of the features for each class. These statistics are used to generate "pseudo-features" that train a lightweight auxiliary classifier to identify the task origin of an input.

3. Task-Adaptive Prediction (TAP)

TAP is the final step of making a classification across all seen classes. By combining the task-specific insights from WTP and the task-identity from TII, the model can make a final, informed prediction. The use of statistical modeling allows the model to "rehearse" in the feature space, effectively simulating old data without needing the original images.

Experimental Results

We evaluated HiDe-Prompt against several benchmarks, including Split CIFAR-100, Split ImageNet-R, and the 5-Datasets mix. We tested across various pre-training paradigms, including supervised (Sup-21K) and SSL methods (iBOT, DINO, and MoCo v3).

Performance Gains

HiDe-Prompt consistently outperformed existing rehearsal-free methods, with the most dramatic improvements seen on SSL backbones:

  • Split CIFAR-100 (iBOT): Achieved 93.48% accuracy, representing a 14.37% lead over CODA-Prompt.
  • Split ImageNet-R: Maintained a lead of up to 9.61% over competitors.
  • Forgetting: The Final Forgetting Measure (FFM) remained near zero in several tests, indicating nearly perfect retention of prior knowledge.

Efficiency

Beyond accuracy, HiDe-Prompt is more resource-efficient. By separating prompt construction from task inference, it requires significantly less GPU memory (e.g., ~16GB vs. ~25GB for CODA-Prompt on CIFAR-100), making it more viable for real-world deployment.

Conclusion and Future Work

HiDe-Prompt addresses the hidden sub-optimality of prompt-based CL by rethinking how task-specific knowledge is organized and retrieved. By decomposing the problem into hierarchical components and utilizing statistical feature rehearsal, it bridges the gap between supervised and self-supervised pre-training performance.

Future research will focus on adapting this framework for non-Transformer architectures and exploring "slow-fast" systems where the backbone can be updated gradually without losing its foundational knowledge. As it stands, HiDe-Prompt provides a robust, efficient, and privacy-preserving path forward for continual learning in the era of massive self-supervised models.

Summary updated

Abstract

Prompt-based continual learning is an emerging direction in leveraging pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical research reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome the exposed sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Following these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our code is available at \url{https://github.com/thu-ml/HiDe-Prompt}.
BibTeX
@inproceedings{wang2023hierarchical,
  author = {Liyuan Wang and Jingyi Xie and Xingxing Zhang and Mingyi Huang and Hang Su and Jun Zhu},
  title = {{Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=9XieH21Tlf},
}
Figure