TL;DR
TEA-Prompt introduces a rehearsal-free continual learning method that preserves a model's behavior in function space rather than just freezing its parameters in weight space. By using a single shared prompt coupled with class-specific "anchoring" statistics, it achieves state-of-the-art accuracy on major benchmarks with a tiny parameter footprint (0.38M).
What Problem Does This Paper Address?
In the world of machine learning, Continual Learning (CL) is the quest to teach a model new tasks sequentially without it "catastrophically forgetting" what it already learned. For Vision Transformers (ViTs), a popular approach is prompt tuning: keeping the massive pre-trained backbone frozen and only training a small set of "prompt tokens" to adapt to new tasks.
The most efficient version of this uses a single shared prompt for all tasks. However, this creates a tug-of-war: if you update the prompt for Task B, you likely break its functionality for Task A. Existing solutions usually try to protect old knowledge in "parameter space"—by freezing specific tokens or using sparse gating. The authors argue this is flawed. Freezing tokens kills "plasticity" (the ability to learn), while sparse gating is often too rigid.
Instead, they ask: What if we let the parameters change, as long as the prompt's behavior (its contribution to the attention mechanism) remains the same for old classes?
Methods
The authors propose TEA-Prompt (Token Evaluation and Anchoring), which consists of two main components:
- Token Evaluation Module: Instead of a hard "on/off" switch for prompt tokens, this module assigns each token a continuous, input-conditional relevance score. It uses a "reduced query" (the average of input queries) to score tokens and passes them through a thresholded sigmoid. This allows for "soft specialization," where tokens can contribute to multiple tasks to varying degrees.
- Attention-Behavior Anchoring Loss ($L_{AD}$): This is the "memory" of the system. After each task, the model stores compact Gaussian statistics (mean and variance) of the queries for each class. When learning a new task, the model samples "pseudo-queries" from these distributions. It then ensures that the current prompt produces the same attention output as a snapshot of the old prompt for those queries.
Crucially, these two are coupled: the anchoring loss is weighted by a token's past relevance. If a token didn't matter for an old class, the model is free to change it aggressively for the new one.
Datasets
The model was evaluated on three primary benchmarks, each partitioned into 10 tasks:
- CIFAR-100: A standard benchmark for general image classification.
- ImageNet-R: A challenging dataset featuring diverse artistic styles (paintings, cartoons, etc.) and high intra-class variance.
- CUB-200: A fine-grained dataset of 200 bird species, testing the model's ability to distinguish subtle differences.
Key Insights and Ideas
- Function Space vs. Weight Space: The core "aha" moment is that we don't need to keep weights static to keep knowledge. By anchoring the output of the attention mechanism, the model maintains stability while retaining the plasticity needed to learn new features.
- Soft Gating is Better than Hard Gating: The authors found that discrete "Top-K" selection is too rigid. Their continuous sigmoid-based weighting allows "lower-ranked" tokens to contribute complementary information, which significantly boosts accuracy.
- Class-Specific > Task-Specific: Storing query statistics per class rather than per task allows for much finer-grained preservation. This prevents the "dilution" of behavior that happens when you aggregate statistics across an entire task.
- Rehearsal-Free Efficiency: Unlike "replay" methods that store actual images from old tasks (which can be a privacy or storage nightmare), TEA-Prompt only stores about 3.7 MB to 8.3 MB of Gaussian statistics.
Key Results
TEA-Prompt consistently outperformed existing prompt-based CL methods across all benchmarks while remaining highly parameter-efficient.
- CIFAR-100: Achieved 90.59% average accuracy, nearly reaching the "Joint Training" upper bound (91.38%) and significantly beating the previous best shared-prompt method, SMoPE (88.86%).
- ImageNet-R: Reached a state-of-the-art 77.40% accuracy with a forgetting rate of only 4.19%.
- CUB-200: Boosted accuracy to 86.84%, a massive jump over other shared-prompt architectures like OVOR (78.11%).
- Efficiency: All of this was achieved with only 0.38M learnable parameters, making it one of the most efficient SOTA models in the field.
Limitations and Future Work
- Linear Scaling of Training Cost: While the memory footprint is small, the computational cost of the anchoring loss grows linearly with the number of seen classes during training. While the authors demonstrate this cost is "cheap" (around 1-2 GFLOPs per step), it could become a bottleneck for sequences with thousands of classes.
- Gaussian Assumption: The method assumes query distributions are Gaussian. While the authors argue this is "mild" due to the averaging effect of the reduced query, more complex distributions might exist in certain domains.
- Future Directions: The authors suggest that this "behavior anchoring" philosophy could be extended beyond prompts to other parameter-efficient fine-tuning methods like LoRA.
Summary updated