One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning

Minh Le, Bao-Ngoc Dao, Huy Nguyen, Quyen Tran, Anh Nguyen, Nhat Ho

TL;DR

SMoPE (Sparse Mixture of Prompt Experts) reconciles a classic trade-off in prompt-based continual learning: the high memory cost of task-specific prompts versus the performance degradation of a single shared prompt. By structuring a single shared prompt into a sparse Mixture of Experts (MoE) architecture, SMoPE achieves state-of-the-art accuracy while slashing parameter counts and reducing computational overhead by up to 50%.

What Problem Does This Paper Address?

In Continual Learning (CL), neural networks must learn a sequence of tasks over time without forgetting previously acquired knowledge—a challenge known as catastrophic forgetting. Recently, prompt-based methods have emerged as a powerful solution, tuning only a small set of learnable parameters (prompts) while keeping the backbone model frozen.

However, current prompt-based methods fall into two flawed paradigms:

  1. Task-Specific Prompts: These allocate a distinct set of prompts for each task. While effective at isolating knowledge, their memory requirements grow linearly with the number of tasks. Furthermore, at inference time, determining which prompt to use requires a costly forward pass through the model to generate a query.
  2. Shared Prompts: These use a single prompt across all tasks. While highly parameter-efficient, continually updating the same parameters causes severe knowledge interference, leading to poor performance.

The authors ask: Can we retain the parameter efficiency of a single shared prompt while achieving the high performance of task-specific approaches?

Methods

The authors introduce SMoPE (Sparse Mixture of Prompt Experts), which treats a single shared prompt as a collection of multiple "prompt experts" within a sparse MoE framework. For any given input, only the top-$K$ most relevant prompt experts are activated and updated.

SMoPE Implementation

To make this architecture viable for continual learning, SMoPE introduces three key innovations:

  • Prompt-Attention Score Aggregation: Standard MoE routing computes a score for every single token, which is computationally expensive. SMoPE instead computes a unified proxy score per expert based on the average token representation ($\tilde{x}$), reducing routing complexity $N$-fold (where $N$ is sequence length).
  • Adaptive Noise Mechanism: To prevent "expert collapse" (where a few experts dominate and get repeatedly overwritten), SMoPE applies a noise penalty specifically to frequently activated experts. This forces the model to explore underutilized experts for new tasks while protecting the knowledge locked inside critical, highly active experts.
  • Prefix Keys as Prototypes: Since past data is unavailable in CL, the authors use the prefix keys from earlier tasks as implicit memory representations (prototypes). A novel prototype-based loss function ($L_{proto}$) ensures that new task training does not distort the input-space specializations learned by experts in past tasks.

Datasets

SMoPE was evaluated on three standard continual learning benchmarks:

  • ImageNet-R: 200 classes with diverse stylistic variations, split into 5, 10, 20, or 50 sequential tasks.
  • CIFAR-100: 100 classes randomly partitioned into 10 sequential tasks.
  • CUB-200: A fine-grained dataset of 200 bird species, divided into 10 tasks.

The authors also tested robustness across different self-supervised pre-training paradigms, including iBOT-1K and DINO-1K.

Key Insights and Ideas

  • Prefix Tuning is Inherently an MoE: The authors build on the mathematical insight that attention heads in Vision Transformers (ViTs) can be viewed as an ensemble of pre-trained experts. Prefix tuning simply injects new experts into this mix.
  • Routing Without Full Forward Passes: Unlike task-specific methods that require forwarding an input through the entire model just to select a prompt key, SMoPE selects experts dynamically within each attention layer using the immediate layer input.
  • Prompts as Data Summaries: Rather than storing raw images or features in a rehearsal buffer, the prompt keys themselves serve as excellent spatial prototypes of past data distributions, enabling rehearsal-free continual learning.

Key Results

SMoPE consistently outperforms both single-prompt and task-specific baselines across all benchmarks while maintaining a tiny parameter footprint.

Main Performance

As shown in the main benchmark results, SMoPE achieves the highest Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA):

Table 1: Performance comparison on ImageNet-R, CIFAR-100, and CUB-200 (10-task splits). ↑indicates higher is better. FAA and CAA are averaged over 5 runs. Bold denotes the best results, excluding joint training.

Robustness to Pre-training Paradigms

SMoPE's architecture generalizes exceptionally well, outperforming state-of-the-art methods even when paired with self-supervised backbones like iBOT and DINO:

Table 2: Performance comparison on ImageNet-R (10-task split) for self-supervised pre-training, iBOT-1K, and DINO-1K. ↑indicates higher is better. FAA and CAA are averaged over 5 runs. Bold highlights the best results.

Efficiency Gains

  • Parameters: SMoPE requires only 0.38M learnable parameters, compared to 4.21M for competitive task-specific methods like HiDe-Prompt and NoRGa.
  • Compute: Because it eliminates the query-generation forward pass, SMoPE reduces both training and testing compute (GFLOPs) by 50% relative to most task-specific prompting baselines.

Limitations and Future Work

While SMoPE successfully mitigates interference using a single shared prompt structure, the authors note that if the number of tasks grows indefinitely, capacity limits might eventually trigger knowledge overwriting.

Future research directions include:

  • Exploring strategies to dynamically expand the prompt length or add more prompt experts as the task sequence scales.
  • Extending the SMoPE framework to other foundational models beyond Vision Transformers, such as Large Language Models (LLMs).

Summary updated

Abstract

Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strategy in this paradigm assigns a dedicated subset of prompts to each task, which, while effective, incurs substantial computational overhead and causes memory requirements to scale linearly with the number of tasks. Conversely, approaches employing a single shared prompt across tasks offer greater efficiency but often suffer from degraded performance due to knowledge interference. To reconcile this trade-off, we propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies. Inspired by recent findings on the relationship between Prefix Tuning and Mixture of Experts (MoE), SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse MoE architecture. For each input, only a select subset of relevant experts is activated, effectively mitigating interference. To facilitate expert selection, we introduce a prompt-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling dynamic and sparse activation. Additionally, we propose an adaptive noise mechanism to encourage balanced expert utilization while preserving knowledge from prior tasks. To further enhance expert specialization, we design a prototype-based loss function that leverages prefix keys as implicit memory representations. Extensive experiments across multiple CL benchmarks demonstrate that SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches, all while significantly reducing parameter counts and computational costs.
BibTeX
@inproceedings{le2026one,
  author = {Minh Le and Bao-Ngoc Dao and Huy Nguyen and Quyen Tran and Anh Nguyen and Nhat Ho},
  title = {{One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=B8eIc9783S},
}
Figure