Abstract
Prompt-based Continual Learning (PCL) has emerged as a state-of-the-art solution for preventing catastrophic forgetting without the privacy risks or memory overhead associated with data buffers. However, existing PCL methods suffer from significant computational burdens due to a two-stage feed-forward process: one for a query Vision Transformer (ViT) to select prompts, and another for a backbone ViT to process the image with those prompts. We introduce OS-Prompt, a one-stage PCL framework that utilizes intermediate layer token embeddings as prompt queries. This eliminates the redundant feed-forward stage, reducing computational costs by ~50% with minimal accuracy loss. Furthermore, we propose a Query-Pool Regularization (QR) loss to enhance representation power during training. Our approach, OS-Prompt++, maintains the 50% inference speedup while outperforming prior two-stage methods by approximately 1.4% across major benchmarks including CIFAR-100, ImageNet-R, and DomainNet.
Introduction
In the field of machine learning, "catastrophic forgetting" remains a primary obstacle to developing models that can learn continuously from a stream of data. While Prompt-based Continual Learning (PCL) has addressed this by keeping a backbone model frozen and learning task-specific prompts, the architecture of current PCL methods is inherently inefficient. Methods like L2P and CodaPrompt rely on a two-stage architecture where an image must pass through a dedicated "Query ViT" just to determine which prompts to retrieve from a pool, before passing through the actual "Backbone ViT" for classification.
This dual-transformer requirement nearly doubles the GFLOPs (Giga Floating Point Operations) and inference latency, making these models impractical for edge devices or real-time applications. This paper presents a streamlined alternative that achieves higher efficiency and superior accuracy.
The One-Stage Framework: OS-Prompt
The core innovation of OS-Prompt is the collapse of the two-stage selection process into a single stage. Instead of using a separate, frozen reference model to generate queries, OS-Prompt leverages the internal representations of the backbone ViT itself.
The Stability Insight
The feasibility of a one-stage approach rests on the observation that the feature space of early layers in a Vision Transformer remains remarkably stable throughout the continual learning process. While deeper layers shift significantly as the model adapts to new tasks, layers 1 through 5 provide a consistent and reliable source for prompt queries. By using token embeddings from these intermediate layers, OS-Prompt removes the need for the initial Query ViT pass entirely.
OS-Prompt++ and Query-Pool Regularization
While intermediate layers are stable, they may lack the "representational power" of the final layers used in two-stage methods. To bridge this gap, we introduced OS-Prompt++, which incorporates a Query-Pool Regularization (QR) loss.
The QR loss functions during the training phase to align the relationship between the intermediate query and the prompt pool with that of a high-quality reference query. Because this loss is only applied during training, it adds no computational overhead to the inference phase. This allows the model to benefit from high-quality prompt selection logic while maintaining the speed of a one-stage architecture during deployment.
Experimental Results
We evaluated OS-Prompt and OS-Prompt++ on three major class-incremental benchmarks: CIFAR-100, ImageNet-R, and DomainNet.
Efficiency Gains
Across various hardware platforms—including NVIDIA A100, V100, and RTX 2080ti—OS-Prompt consistently reduced GFLOPs and GPU latency by approximately 50%. This makes PCL viable for latency-sensitive applications where it was previously too slow.
Accuracy Improvements
Despite the reduction in complexity, OS-Prompt++ demonstrated superior performance compared to existing two-stage state-of-the-art (SOTA) methods:
- ImageNet-R: OS-Prompt++ outperformed CodaPrompt by approximately 1.4%.
- Robustness: The method proved effective even when using unsupervised pre-trained weights (e.g., DINO), indicating that the efficiency gains are not dependent on specific supervised pre-training regimes.
Conclusion
OS-Prompt represents a significant step forward in making continual learning practical for real-world deployment. By recognizing that the backbone model's early layers already contain the information necessary for prompt selection, we can eliminate redundant computations. The addition of the QR loss ensures that this efficiency does not come at the cost of accuracy, but rather enhances it. Future work may explore "early exit" strategies to further reduce the computational footprint of these models without sacrificing their ability to learn indefinitely.
Summary updated