All You Need is One: Capsule Prompt Tuning with a Single Vector

Yiyang Liu, James Chenhao Liang, Heng Fan, Wenhao Yang, Yiming Cui, Xiaotian Han, Lifu Huang, Dongfang Liu, Qifan Wang, Cheng Han

Abstract

Prompt-based learning has emerged as a leading parameter-efficient fine-tuning (PEFT) approach, enabling Large Language Models (LLMs) to adapt to downstream tasks through task-aware guidance. However, traditional prompt tuning faces two significant hurdles: the necessity of exhaustive grid searches to determine optimal prompt length and a lack of instance-specific awareness, which often results in "lonely" prompts that fail to interact effectively with input text. We introduce Capsule Prompt-Tuning (CaPT), an efficient solution that integrates instance-aware semantics into the prompt-tuning process. By utilizing a single "capsule" vector per layer, CaPT acts as an "attention anchor," preserving critical structural information and enhancing interaction across the sequence. Our empirical results demonstrate that CaPT achieves superior performance—reaching 84.03% average accuracy on T5-Large—while utilizing as little as 0.003% of model parameters and reducing training time by nearly 9x.

Introduction

As Large Language Models (LLMs) grow in scale, full fine-tuning becomes computationally prohibitive. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Prompt Tuning, offer a solution by prepending learnable "soft prompts" to the input. Despite their efficiency, these methods are often hindered by a "task-aware but instance-blind" design. Standard soft prompts tend to attend primarily to themselves rather than the input sequence, limiting their ability to guide the model based on specific context. Furthermore, the performance of these models is highly sensitive to prompt length, requiring researchers to perform costly grid searches for every new task.

This paper introduces Capsule Prompt-Tuning (CaPT), a method that replaces long sequences of static prompts with a single, dynamic vector. This approach not only eliminates the need for length-tuning but also significantly improves the model's internal attention mechanisms.

The "Lonely Prompt" and the Attention Anchor

Our preliminary analysis revealed a striking phenomenon: in standard prompt tuning, the soft prompts often exhibit a "self-isolation" behavior. Because they are static across all instances of a task, they fail to engage with the unique semantics of individual inputs.

However, we discovered that incorporating instance-aware information—even in a training-free setting—can stabilize the model's focus. We call this the "Attention Anchor" effect. When instance-specific tokens are placed at the beginning of a sequence, they help the model maintain strong attention on critical structural elements (such as punctuation or template keywords) that are otherwise ignored. This observation serves as the foundation for the CaPT architecture.

Methodology: Capsule Prompt-Tuning (CaPT)

CaPT simplifies the prompt architecture by using exactly one "capsule" vector per Transformer layer. This capsule is a hybrid construction consisting of two primary components:

  1. Task-Aware Component: A single learnable vector that captures the general requirements of the downstream task.
  2. Instance-Aware Component: A dynamic representation derived from the current input sequence (typically the mean-pooled hidden states of the input or the previous layer).

By fusing these components (primarily through addition), the prompt becomes dynamic, adapting to every specific sentence the model processes. This single vector is then either prepended to the input sequence or added directly to the hidden states. This design ensures that the prompt is always relevant to the current context, effectively anchoring the model's attention throughout the deep layers of the network.

Experimental Setup

We evaluated CaPT across a diverse range of Natural Language Understanding (NLU) tasks using the SuperGLUE benchmark, including:

  • Question Answering: BoolQ and MultiRC.
  • Natural Language Inference: CB and RTE.
  • Reasoning and Disambiguation: COPA and WiC.

The method was tested on several prominent architectures, including T5 (Base and Large), Llama-3.2-1B, and Qwen-2.5-1.5B, to ensure the findings generalized across both encoder-decoder and decoder-only models.

Key Results

Performance and Efficiency

CaPT consistently outperformed traditional prompt tuning and matched or exceeded full fine-tuning results. On T5-Large, CaPT achieved an average accuracy of 84.03%, compared to 78.16% for standard prompt tuning and 83.60% for full fine-tuning.

In terms of parameter efficiency, the gains were even more pronounced. For the Llama-3.2-1B model, CaPT required tuning only 0.003% of the total parameters. Because the method uses a fixed length of one vector, it completely bypasses the need for prompt-length grid searches, resulting in a 8.7x speedup in training time compared to traditional methods.

The Power of Instance Awareness

Our experiments confirmed that LLMs are "hungry" for instance-specific guidance. Even without additional training, prepending a mean-pooled version of the input to a pre-trained model improved performance. CaPT leverages this hunger by formalizing the instance-aware component within the learnable framework, ensuring the "Attention Anchor" remains robust across all layers.

Conclusion

Capsule Prompt-Tuning (CaPT) demonstrates that in the realm of prompt engineering, quality and relevance outweigh quantity. By replacing a sequence of static vectors with a single, instance-aware capsule, we can achieve higher accuracy with significantly lower computational overhead. CaPT not only solves the practical issue of prompt-length optimization but also provides a more theoretically sound way for prompts to interact with input data. Future work will explore the application of the "Attention Anchor" effect in multi-modal settings and long-form generative tasks.

Summary updated

Abstract

Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i.e., one single capsule prompt). Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e.g., 84.03\% average accuracy on T5-Large), serving as an "attention anchor," while enjoying high parameter efficiency (e.g., 0.003\% of model parameters on Llama3.2-1B).
BibTeX
@inproceedings{liu2025all,
  author = {Yiyang Liu and James Chenhao Liang and Heng Fan and Wenhao Yang and Yiming Cui and Xiaotian Han and Lifu Huang and Dongfang Liu and Qifan Wang and Cheng Han},
  title = {{All You Need is One: Capsule Prompt Tuning with a Single Vector}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=WZS7sJiSUm},
}
Figure