Abstract
Continual Learning (CL) requires models to learn from a sequential stream of tasks while mitigating catastrophic forgetting. While prompt tuning has emerged as a leading rehearsal-free approach, existing methods rely on static strategies—either sharing a single universal prompt or assigning unique prompts to every task. These approaches fail to account for the diverse semantic shifts found in real-world data. We present AdaPromptCL, an adaptive prompt tuning framework that utilizes an "assign-and-refine" strategy. By dynamically grouping tasks based on semantic similarity, AdaPromptCL optimizes the balance between knowledge transfer and forgetting prevention. Our method outperforms state-of-the-art baselines by up to 21.3% in complex scenarios with varying semantic shifts.
Introduction: The Challenge of Semantic Shifting
In Continual Learning, the primary goal is to maintain performance on old tasks while acquiring new knowledge. Recent advancements have leveraged "prompt tuning," where small sets of learnable parameters are added to frozen pre-trained models. However, current prompt-based methods typically fall into two rigid categories:
- Universal Prompting: A single set of prompts is shared across all tasks. This promotes positive knowledge transfer when tasks are similar (mild shifts) but leads to catastrophic forgetting when tasks are divergent (abrupt shifts).
- Specific Prompting: Each task is assigned its own unique prompt. While this effectively isolates tasks to prevent forgetting, it ignores potential synergies between similar tasks and leads to linear parameter growth.
Real-world data streams are rarely uniform; they often contain a mixture of mild and abrupt semantic shifts. AdaPromptCL addresses this by deciding on-the-fly when to share a prompt and when to initialize a new one, adapting to the specific degree of semantic shifting in the task stream.
Methodology: The Assign-and-Refine Strategy
AdaPromptCL introduces a novel semantic grouping mechanism designed to handle unpredictable task sequences. The framework operates in two primary stages:
Task Semantic Extraction
Before training begins on a new task, the model undergoes a brief "warm-up" phase. This phase generates a semantic representation of the incoming task, which serves as the basis for grouping decisions.
Step I: Assign
The model evaluates the distance between the new task's semantic representation and existing task groups. If the distance is below a predefined threshold ($R$), the task is assigned to the most similar existing group, sharing its prompt. If the task is sufficiently distinct, a new prompt group is initialized.
Step II: Refine
Online clustering is sensitive to task order, which can lead to sub-optimal "greedy" groupings early in the stream. To mitigate this, AdaPromptCL proactively collects prospective prompts—alternative groupings that might become more efficient as more tasks arrive. When a new task provides evidence that a more compact or accurate grouping is possible (e.g., merging multiple redundant groups), the model refines the groups and retrieves the corresponding pre-trained prospective prompts.
Experimental Results
We evaluated AdaPromptCL across various scenarios, including uniformly mild shifts (CIFAR-100), uniformly abrupt shifts (VTAB-19T), and complex "varying" shifts (VTAB-SimS and VTAB-RecR).
Performance and Efficiency
As shown in the results, AdaPromptCL achieves a superior balance across all scenarios. In the "Varying" shift benchmarks—the most representative of real-world conditions—AdaPromptCL outperformed the best baselines by an average of 6.35% to 6.91%.
Beyond accuracy, the method is highly parameter-efficient. On the VTAB-Rec10 benchmark, AdaPromptCL achieved 13.6% higher accuracy than specific prompting methods while using only 12.4% of the total number of prompts. This demonstrates that adaptive grouping not only improves performance through positive transfer but also significantly reduces the model's memory footprint.
Conclusion
AdaPromptCL demonstrates that the binary choice between universal and specific prompting is a false dichotomy. By employing an adaptive "assign-and-refine" mechanism, models can navigate the complexities of real-world semantic shifts, achieving state-of-the-art performance with high efficiency. Future work will focus on automating threshold selection and extending this adaptive logic to multi-modal and natural language processing domains.
Summary updated