Anchor Behavior, Not Parameters: Rehearsal-Free Continual Learning with a Single Shared Prompt

Duy-Cuong Bui, Trung-Anh Dang, Vincent Nguyen, Duc V. Le, Ngoc-Son VuarXiv 2026 · arXiv

TL;DR

TEA-Prompt introduces a rehearsal-free continual learning method that preserves a model's behavior in function space rather than just freezing its parameters in weight space. By using a single shared prompt coupled with class-specific "anchoring" statistics, it achieves state-of-the-art accuracy on major benchmarks with a tiny parameter footprint (0.38M).

What Problem Does This Paper Address?

In the world of machine learning, Continual Learning (CL) is the quest to teach a model new tasks sequentially without it "catastrophically forgetting" what it already learned. For Vision Transformers (ViTs), a popular approach is prompt tuning: keeping the massive pre-trained backbone frozen and only training a small set of "prompt tokens" to adapt to new tasks.

The most efficient version of this uses a single shared prompt for all tasks. However, this creates a tug-of-war: if you update the prompt for Task B, you likely break its functionality for Task A. Existing solutions usually try to protect old knowledge in "parameter space"—by freezing specific tokens or using sparse gating. The authors argue this is flawed. Freezing tokens kills "plasticity" (the ability to learn), while sparse gating is often too rigid.

Instead, they ask: What if we let the parameters change, as long as the prompt's behavior (its contribution to the attention mechanism) remains the same for old classes?

Methods

The authors propose TEA-Prompt (Token Evaluation and Anchoring), which consists of two main components:

  1. Token Evaluation Module: Instead of a hard "on/off" switch for prompt tokens, this module assigns each token a continuous, input-conditional relevance score. It uses a "reduced query" (the average of input queries) to score tokens and passes them through a thresholded sigmoid. This allows for "soft specialization," where tokens can contribute to multiple tasks to varying degrees.
  2. Attention-Behavior Anchoring Loss ($L_{AD}$): This is the "memory" of the system. After each task, the model stores compact Gaussian statistics (mean and variance) of the queries for each class. When learning a new task, the model samples "pseudo-queries" from these distributions. It then ensures that the current prompt produces the same attention output as a snapshot of the old prompt for those queries.

Crucially, these two are coupled: the anchoring loss is weighted by a token's past relevance. If a token didn't matter for an old class, the model is free to change it aggressively for the new one.

Figure 1. Overview of TEA-Prompt. Left: Inside each MSA layer of the frozen ViT, prompt tokens (PK, PV ) are prepended to the frozen keys and values. The reduced query scores every prompt token (Eq. (2)); a thresholded sigmoid turns the scores into evaluation weights (Eq. (3)) whose logarithm shifts the prompt attention logits Lprompt (Eq. (4)), modulating each token’s attention mass before the joint softmax with the input-token logits Lpre-trained (Eq. (5)). Right: At the end of task t−1 the prompt is snapshotted and class- conditional Gaussian query statistics are stored (Eq. (6)); while learning task t, queries sampled from these statistics (Eq. (7)) probe the prompt attention of the frozen snapshot (A(t−1)) and of the current prompt (A(t)), and LAD (Eq. (9)) anchors the current behavior to the old one, weighted by each token’s past relevance.

Datasets

The model was evaluated on three primary benchmarks, each partitioned into 10 tasks:

  • CIFAR-100: A standard benchmark for general image classification.
  • ImageNet-R: A challenging dataset featuring diverse artistic styles (paintings, cartoons, etc.) and high intra-class variance.
  • CUB-200: A fine-grained dataset of 200 bird species, testing the model's ability to distinguish subtle differences.

Key Insights and Ideas

  • Function Space vs. Weight Space: The core "aha" moment is that we don't need to keep weights static to keep knowledge. By anchoring the output of the attention mechanism, the model maintains stability while retaining the plasticity needed to learn new features.
  • Soft Gating is Better than Hard Gating: The authors found that discrete "Top-K" selection is too rigid. Their continuous sigmoid-based weighting allows "lower-ranked" tokens to contribute complementary information, which significantly boosts accuracy.
  • Class-Specific > Task-Specific: Storing query statistics per class rather than per task allows for much finer-grained preservation. This prevents the "dilution" of behavior that happens when you aggregate statistics across an entire task.
  • Rehearsal-Free Efficiency: Unlike "replay" methods that store actual images from old tasks (which can be a privacy or storage nightmare), TEA-Prompt only stores about 3.7 MB to 8.3 MB of Gaussian statistics.

Key Results

TEA-Prompt consistently outperformed existing prompt-based CL methods across all benchmarks while remaining highly parameter-efficient.

  • CIFAR-100: Achieved 90.59% average accuracy, nearly reaching the "Joint Training" upper bound (91.38%) and significantly beating the previous best shared-prompt method, SMoPE (88.86%).
  • ImageNet-R: Reached a state-of-the-art 77.40% accuracy with a forgetting rate of only 4.19%.
  • CUB-200: Boosted accuracy to 86.84%, a massive jump over other shared-prompt architectures like OVOR (78.11%).
  • Efficiency: All of this was achieved with only 0.38M learnable parameters, making it one of the most efficient SOTA models in the field.
Table 1. Performance comparison on CIFAR-100, ImageNet-R, and CUB-200 datasets. Params counts the learnable parameters on ImageNet-R with 10 tasks. ( ”-” means Not Available)

Limitations and Future Work

  • Linear Scaling of Training Cost: While the memory footprint is small, the computational cost of the anchoring loss grows linearly with the number of seen classes during training. While the authors demonstrate this cost is "cheap" (around 1-2 GFLOPs per step), it could become a bottleneck for sequences with thousands of classes.
  • Gaussian Assumption: The method assumes query distributions are Gaussian. While the authors argue this is "mild" due to the averaging effect of the reduced query, more complex distributions might exist in certain domains.
  • Future Directions: The authors suggest that this "behavior anchoring" philosophy could be extended beyond prompts to other parameter-efficient fine-tuning methods like LoRA.

Summary updated

Abstract

Prompt-based continual learning adapts a frozen pre-trained vision transformer by tuning a small set of prompt tokens. Recently, one-prompt methods which share one single prompt across all tasks make this paradigm maximally parameter efficient. The open question is how a shared prompt can keep the knowledge of old classes while being updated for new ones. Existing methods protect old knowledge in parameter space, where they freeze task-specific prompts, expand new components, or sparsely gate which tokens receive gradients. However, this method sacrifices plasticity, yet the drift it still permits in shared tokens can corrupt the prompt's function on past classes. What must be preserved is the prompt's behavior. We propose TEA-Prompt, which couples (i) a token evaluation module that assigns each prompt token a continuous, input-conditional relevance score injected into the attention logits, softly specializing tokens across tasks without discarding any of them, and (ii) an attention-behavior anchoring loss that constrains each token's attention output around class-specific query regions using only compact Gaussian query statistics without stored old images, weighted by the token's past relevance. Anchoring the attention value allows the parameters to continue updating freely, as long as their interaction remains unchanged. TEA-Prompt achieves state-of-the-art accuracy with markedly lower forgetting on CIFAR-100, ImageNet-R, and CUB-200 with only 0.38M parameters. Code is publicy available at https://github.com/cuonga1cvp/TEA-Prompt.git
BibTeX
@misc{bui2026anchor,
  author = {Duy-Cuong Bui and Trung-Anh Dang and Vincent Nguyen and Duc V. Le and Ngoc-Son Vu},
  title = {{Anchor Behavior, Not Parameters: Rehearsal-Free Continual Learning with a Single Shared Prompt}},
  year = {2026},
  eprint = {2611.12777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12777},
}
Figure