One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning

Doyoung Kim, Susik Yoon, Dongmin Park, Youngjun Lee, Hwanjun Song, Jihwan Bang, Jae-Gil Lee

Abstract

Continual Learning (CL) requires models to learn from a sequential stream of tasks while mitigating catastrophic forgetting. While prompt tuning has emerged as a leading rehearsal-free approach, existing methods rely on static strategies—either sharing a single universal prompt or assigning unique prompts to every task. These approaches fail to account for the diverse semantic shifts found in real-world data. We present AdaPromptCL, an adaptive prompt tuning framework that utilizes an "assign-and-refine" strategy. By dynamically grouping tasks based on semantic similarity, AdaPromptCL optimizes the balance between knowledge transfer and forgetting prevention. Our method outperforms state-of-the-art baselines by up to 21.3% in complex scenarios with varying semantic shifts.

Introduction: The Challenge of Semantic Shifting

In Continual Learning, the primary goal is to maintain performance on old tasks while acquiring new knowledge. Recent advancements have leveraged "prompt tuning," where small sets of learnable parameters are added to frozen pre-trained models. However, current prompt-based methods typically fall into two rigid categories:

  1. Universal Prompting: A single set of prompts is shared across all tasks. This promotes positive knowledge transfer when tasks are similar (mild shifts) but leads to catastrophic forgetting when tasks are divergent (abrupt shifts).
  2. Specific Prompting: Each task is assigned its own unique prompt. While this effectively isolates tasks to prevent forgetting, it ignores potential synergies between similar tasks and leads to linear parameter growth.

Real-world data streams are rarely uniform; they often contain a mixture of mild and abrupt semantic shifts. AdaPromptCL addresses this by deciding on-the-fly when to share a prompt and when to initialize a new one, adapting to the specific degree of semantic shifting in the task stream.

Conceptual comparison of prompt pool strategies (Universal, Specific, and Adaptive) and grouping mechanisms.

Methodology: The Assign-and-Refine Strategy

AdaPromptCL introduces a novel semantic grouping mechanism designed to handle unpredictable task sequences. The framework operates in two primary stages:

Task Semantic Extraction

Before training begins on a new task, the model undergoes a brief "warm-up" phase. This phase generates a semantic representation of the incoming task, which serves as the basis for grouping decisions.

Step I: Assign

The model evaluates the distance between the new task's semantic representation and existing task groups. If the distance is below a predefined threshold ($R$), the task is assigned to the most similar existing group, sharing its prompt. If the task is sufficiently distinct, a new prompt group is initialized.

Step II: Refine

Online clustering is sensitive to task order, which can lead to sub-optimal "greedy" groupings early in the stream. To mitigate this, AdaPromptCL proactively collects prospective prompts—alternative groupings that might become more efficient as more tasks arrive. When a new task provides evidence that a more compact or accurate grouping is possible (e.g., merging multiple redundant groups), the model refines the groups and retrieves the corresponding pre-trained prospective prompts.

Experimental Results

We evaluated AdaPromptCL across various scenarios, including uniformly mild shifts (CIFAR-100), uniformly abrupt shifts (VTAB-19T), and complex "varying" shifts (VTAB-SimS and VTAB-RecR).

Accuracy comparison of AdaPromptCL (Adaptive) against Universal (LAE) and Specific (S-Prompts) baselines across different semantic shift scenarios.

Performance and Efficiency

As shown in the results, AdaPromptCL achieves a superior balance across all scenarios. In the "Varying" shift benchmarks—the most representative of real-world conditions—AdaPromptCL outperformed the best baselines by an average of 6.35% to 6.91%.

Beyond accuracy, the method is highly parameter-efficient. On the VTAB-Rec10 benchmark, AdaPromptCL achieved 13.6% higher accuracy than specific prompting methods while using only 12.4% of the total number of prompts. This demonstrates that adaptive grouping not only improves performance through positive transfer but also significantly reduces the model's memory footprint.

Conclusion

AdaPromptCL demonstrates that the binary choice between universal and specific prompting is a false dichotomy. By employing an adaptive "assign-and-refine" mechanism, models can navigate the complexities of real-world semantic shifts, achieving state-of-the-art performance with high efficiency. Future work will focus on automating threshold selection and extending this adaptive logic to multi-modal and natural language processing domains.

Summary updated

BibTeX
@inproceedings{kim2024one,
  author = {Doyoung Kim and Susik Yoon and Dongmin Park and Youngjun Lee and Hwanjun Song and Jihwan Bang and Jae-Gil Lee},
  title = {{One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=WUi1AqhKn5},
}
Figure