One-stage Prompt-based Continual Learning

Youngeun Kim, Yuhang Li, Priyadarshini Panda

Abstract

Prompt-based Continual Learning (PCL) has emerged as a state-of-the-art solution for preventing catastrophic forgetting without the privacy risks or memory overhead associated with data buffers. However, existing PCL methods suffer from significant computational burdens due to a two-stage feed-forward process: one for a query Vision Transformer (ViT) to select prompts, and another for a backbone ViT to process the image with those prompts. We introduce OS-Prompt, a one-stage PCL framework that utilizes intermediate layer token embeddings as prompt queries. This eliminates the redundant feed-forward stage, reducing computational costs by ~50% with minimal accuracy loss. Furthermore, we propose a Query-Pool Regularization (QR) loss to enhance representation power during training. Our approach, OS-Prompt++, maintains the 50% inference speedup while outperforming prior two-stage methods by approximately 1.4% across major benchmarks including CIFAR-100, ImageNet-R, and DomainNet.

Introduction

In the field of machine learning, "catastrophic forgetting" remains a primary obstacle to developing models that can learn continuously from a stream of data. While Prompt-based Continual Learning (PCL) has addressed this by keeping a backbone model frozen and learning task-specific prompts, the architecture of current PCL methods is inherently inefficient. Methods like L2P and CodaPrompt rely on a two-stage architecture where an image must pass through a dedicated "Query ViT" just to determine which prompts to retrieve from a pool, before passing through the actual "Backbone ViT" for classification.

This dual-transformer requirement nearly doubles the GFLOPs (Giga Floating Point Operations) and inference latency, making these models impractical for edge devices or real-time applications. This paper presents a streamlined alternative that achieves higher efficiency and superior accuracy.

The One-Stage Framework: OS-Prompt

The core innovation of OS-Prompt is the collapse of the two-stage selection process into a single stage. Instead of using a separate, frozen reference model to generate queries, OS-Prompt leverages the internal representations of the backbone ViT itself.

The Stability Insight

The feasibility of a one-stage approach rests on the observation that the feature space of early layers in a Vision Transformer remains remarkably stable throughout the continual learning process. While deeper layers shift significantly as the model adapts to new tasks, layers 1 through 5 provide a consistent and reliable source for prompt queries. By using token embeddings from these intermediate layers, OS-Prompt removes the need for the initial Query ViT pass entirely.

OS-Prompt++ and Query-Pool Regularization

While intermediate layers are stable, they may lack the "representational power" of the final layers used in two-stage methods. To bridge this gap, we introduced OS-Prompt++, which incorporates a Query-Pool Regularization (QR) loss.

The QR loss functions during the training phase to align the relationship between the intermediate query and the prompt pool with that of a high-quality reference query. Because this loss is only applied during training, it adds no computational overhead to the inference phase. This allows the model to benefit from high-quality prompt selection logic while maintaining the speed of a one-stage architecture during deployment.

Experimental Results

We evaluated OS-Prompt and OS-Prompt++ on three major class-incremental benchmarks: CIFAR-100, ImageNet-R, and DomainNet.

Efficiency Gains

Across various hardware platforms—including NVIDIA A100, V100, and RTX 2080ti—OS-Prompt consistently reduced GFLOPs and GPU latency by approximately 50%. This makes PCL viable for latency-sensitive applications where it was previously too slow.

Accuracy Improvements

Despite the reduction in complexity, OS-Prompt++ demonstrated superior performance compared to existing two-stage state-of-the-art (SOTA) methods:

  • ImageNet-R: OS-Prompt++ outperformed CodaPrompt by approximately 1.4%.
  • Robustness: The method proved effective even when using unsupervised pre-trained weights (e.g., DINO), indicating that the efficiency gains are not dependent on specific supervised pre-training regimes.

Conclusion

OS-Prompt represents a significant step forward in making continual learning practical for real-world deployment. By recognizing that the backbone model's early layers already contain the information necessary for prompt selection, we can eliminate redundant computations. The addition of the QR loss ensures that this efficiency does not come at the cost of accuracy, but rather enhances it. Future work may explore "early exit" strategies to further reduce the computational footprint of these models without sacrificing their ability to learn indefinitely.

Summary updated

Abstract

Prompt-based Continual Learning (PCL) has gained considerable attention as a promising continual learning solution as it achieves state-of-the-art performance while preventing privacy violation and memory overhead issues. Nonetheless, existing PCL approaches face significant computational burdens because of two Vision Transformer (ViT) feed-forward stages; one is for the query ViT that generates a prompt query to select prompts inside a prompt pool; the other one is a backbone ViT that mixes information between selected prompts and image tokens. To address this, we introduce a one-stage PCL framework by directly using the intermediate layer's token embedding as a prompt query. This design removes the need for an additional feed-forward stage for query ViT, resulting in ~50% computational cost reduction for both training and inference with marginal accuracy drop < 1%. We further introduce a Query-Pool Regularization (QR) loss that regulates the relationship between the prompt query and the prompt pool to improve representation power. The QR loss is only applied during training time, so there is no computational overhead at inference from the QR loss. With the QR loss, our approach maintains ~ 50% computational cost reduction during inference as well as outperforms the prior two-stage PCL methods by ~1.4% on public class-incremental continual learning benchmarks including CIFAR-100, ImageNet-R, and DomainNet.
BibTeX
@misc{kim2024one,
  author = {Youngeun Kim and Yuhang Li and Priyadarshini Panda},
  title = {{One-stage Prompt-based Continual Learning}},
  year = {2024},
  eprint = {2402.16189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16189},
}
Figure