Robust Task Selection for Prompt Based Continual Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 122 papers29 conceptsfrozen 2026-10-04

In the field of continual learning, the goal is to enable models to learn a sequence of tasks without "forgetting" previous ones. Recent research has shifted away from storing old data (rehearsal) toward using frozen pre-trained models—like Vision Transformers (ViT) or Large Language Models (LLMs)—and adding small, task-specific parameters called prompts or adapters.

However, a critical bottleneck has emerged: even if a model successfully learns a prompt for Task A and a prompt for Task B, it often fails at test time because it cannot decide which prompt to use. This "selection failure" is often more damaging than the actual forgetting of the task knowledge itself.

The Retrieval Bottleneck

Traditional prompt-based methods, such as L2P or DualPrompt, use a "prompt pool" where each task is associated with a specific key. At inference, the model generates a query from the input image and looks up the most similar key to select the corresponding prompt.

As Martin Menabue and colleagues point out in STAR-Prompt, this selection mechanism is itself prone to catastrophic forgetting. As the model learns new tasks, the "keys" in the prompt pool drift or become crowded. If the model selects the wrong prompt, it applies the wrong context to the input, leading to a total failure in classification—even if the correct prompt is still perfectly preserved in memory. Muhammad Rifki Kurniawan describes this in EvoPrompt as a "fragile all-or-nothing lookup process" that lacks the robustness needed for complex, sequential environments.

Semantic Anchoring and Image Tokens

To solve this, researchers are moving toward semantic prompting, which replaces arbitrary keys with stable, meaningful features. Instead of trying to "predict" a task ID, these methods use the inherent information within the input to guide prompt selection.

In I-Prompt, Jisu Han and Jaemin Na leverage the semantic information already present in the image tokens of a Vision Transformer. By integrating prompt selection into a single forward pass, they eliminate the need for a separate, error-prone task-prediction step. Similarly, Baocai Yin in AESP uses class-level semantic "hints" (like text-based prototypes from CLIP) to anchor the prompts, ensuring the model stays on track even as the number of tasks grows.

From Discrete Pools to Continuous Generation

Another strategy to bypass the selection bottleneck is to move away from discrete "pools" entirely. If the model doesn't have to pick "Prompt 1" or "Prompt 2," it can't pick the wrong one.

  • Instance-Specific Generation: Muhammad Rifki Kurniawan proposes EvoPrompt, which uses a neural bottleneck (an MLP) to generate a unique prompt for every single instance. This makes the system more robust because it relies on a continuous mapping rather than a discrete index.
  • Shared Attention: In the realm of LLMs, Weixiang Zhao introduced SAPT, which uses a shared attention mechanism to combine different parameter blocks. Instead of a hard "routing" to one expert, it uses instance-level attention to softly weight the contribution of different learned modules.

Aligning Training and Inference

A final piece of the puzzle is the mismatch between how these models are trained and how they are tested. During training, we usually know which task an image belongs to, so we use the "correct" prompt. At test time, we don't.

Zhanxin Gao addresses this in CPrompt by aligning the classifiers and prompts. The intuition is that the classifier should be trained to be "prompt-resilient"—it should still provide a reasonable answer even if the prompt selection mechanism is slightly off. By forcing this consistency during training, the model becomes much more reliable during the "unconstrained" selection process at inference.

By focusing on the retrieval and selection of knowledge rather than just its storage, this research direction is closing the gap between experimental continual learning and real-world applications where task boundaries are often blurred or unknown.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_task_selection_for_prompt_based_c_c321a058,
  title = {Robust Task Selection for Prompt Based Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-task-selection-for-prompt-based-continual-learning/rdr_c321a058c8014b3ca5ee0a652e299c8e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_task_selection_for_prompt_based_c_c321a058_v1,
  title = {Robust Task Selection for Prompt Based Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-task-selection-for-prompt-based-continual-learning/rdr_c321a058c8014b3ca5ee0a652e299c8e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}