Abstract
Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strategy in this paradigm assigns a dedicated subset of prompts to each task, which, while effective, incurs substantial computational overhead and causes memory requirements to scale linearly with the number of tasks. Conversely, approaches employing a single shared prompt across tasks offer greater efficiency but often suffer from degraded performance due to knowledge interference. To reconcile this trade-off, we propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies. Inspired by recent findings on the relationship between Prefix Tuning and Mixture of Experts (MoE), SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse MoE architecture. For each input, only a select subset of relevant experts is activated, effectively mitigating interference. To facilitate expert selection, we introduce a prompt-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling dynamic and sparse activation. Additionally, we propose an adaptive noise mechanism to encourage balanced expert utilization while preserving knowledge from prior tasks. To further enhance expert specialization, we design a prototype-based loss function that leverages prefix keys as implicit memory representations. Extensive experiments across multiple CL benchmarks demonstrate that SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches, all while significantly reducing parameter counts and computational costs.
Introduction
In Continual Learning (CL), neural networks must learn a sequence of tasks without forgetting previous ones—a challenge known as "catastrophic forgetting." Recently, prompt-based methods have become popular by keeping a large pre-trained model (like a Vision Transformer) frozen and only training a small set of "prompts" to guide the model for each new task. However, researchers face a frustrating trade-off between Task-Specific Prompts, which scale poorly as parameters grow linearly with tasks, and Shared Prompts, which are efficient but suffer from knowledge interference. SMoPE (Sparse Mixture of Prompt Experts) addresses this by keeping the efficiency of a single shared prompt while achieving the performance of task-specific approaches.
The SMoPE Framework
SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse Mixture of Experts (MoE) architecture. Instead of treating a prompt as a monolithic block, the model dynamically selects a subset of relevant experts for each input.
Key technical innovations include:
- Prompt-Attention Score Aggregation: Standard MoE models compute a score for every single token, which is computationally expensive. SMoPE computes a unified "proxy score" for each expert based on the average representation of the input, resulting in an $N$-fold reduction in complexity.
- Sparse Activation: For any given input, only the top-$K$ most relevant experts are activated. This creates implicit parameter partitioning, allowing different experts to specialize in different data types without explicit task labels.
- Adaptive Noise Mechanism: To prevent over-reliance on a few "super-experts," noise is added to the scores of frequently used experts. This encourages the model to explore underutilized experts for new tasks while protecting existing knowledge.
- Prototype-based Loss: Prefix keys from previous tasks are treated as "prototypes." Regularizing current training against these prototypes helps the model remember the input space regions that previous experts were responsible for.
Experimental Results
The framework was evaluated on three standard benchmarks: ImageNet-R, CIFAR-100, and CUB-200. SMoPE consistently outperformed task-specific methods and maintained a lead even in grueling 50-task splits.
As shown in the results, SMoPE achieved a Final Average Accuracy (FAA) of 79.32% on ImageNet-R, beating the previous best method (VQ-Prompt). Beyond accuracy, SMoPE offers massive efficiency gains, reducing learnable parameters from ~4.2M to 0.38M and cutting inference/training GFLOPs by 50% compared to methods requiring pre-computed queries.
Conclusion
SMoPE demonstrates that a "shared but sparse" approach can reconcile the efficiency-performance trade-off in prompt-based continual learning. By leveraging the mathematical connection between prefix tuning and MoE, the model achieves state-of-the-art results with significantly lower computational overhead. Future work will explore extending this logic to other foundational models like LLMs and implementing dynamic expert addition for indefinite task sequences.
Summary updated