Abstract
Prompt-based learning has emerged as a leading parameter-efficient fine-tuning (PEFT) approach, enabling Large Language Models (LLMs) to adapt to downstream tasks through task-aware guidance. However, traditional prompt tuning faces two significant hurdles: the necessity of exhaustive grid searches to determine optimal prompt length and a lack of instance-specific awareness, which often results in "lonely" prompts that fail to interact effectively with input text. We introduce Capsule Prompt-Tuning (CaPT), an efficient solution that integrates instance-aware semantics into the prompt-tuning process. By utilizing a single "capsule" vector per layer, CaPT acts as an "attention anchor," preserving critical structural information and enhancing interaction across the sequence. Our empirical results demonstrate that CaPT achieves superior performance—reaching 84.03% average accuracy on T5-Large—while utilizing as little as 0.003% of model parameters and reducing training time by nearly 9x.
Introduction
As Large Language Models (LLMs) grow in scale, full fine-tuning becomes computationally prohibitive. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Prompt Tuning, offer a solution by prepending learnable "soft prompts" to the input. Despite their efficiency, these methods are often hindered by a "task-aware but instance-blind" design. Standard soft prompts tend to attend primarily to themselves rather than the input sequence, limiting their ability to guide the model based on specific context. Furthermore, the performance of these models is highly sensitive to prompt length, requiring researchers to perform costly grid searches for every new task.
This paper introduces Capsule Prompt-Tuning (CaPT), a method that replaces long sequences of static prompts with a single, dynamic vector. This approach not only eliminates the need for length-tuning but also significantly improves the model's internal attention mechanisms.
The "Lonely Prompt" and the Attention Anchor
Our preliminary analysis revealed a striking phenomenon: in standard prompt tuning, the soft prompts often exhibit a "self-isolation" behavior. Because they are static across all instances of a task, they fail to engage with the unique semantics of individual inputs.
However, we discovered that incorporating instance-aware information—even in a training-free setting—can stabilize the model's focus. We call this the "Attention Anchor" effect. When instance-specific tokens are placed at the beginning of a sequence, they help the model maintain strong attention on critical structural elements (such as punctuation or template keywords) that are otherwise ignored. This observation serves as the foundation for the CaPT architecture.
Methodology: Capsule Prompt-Tuning (CaPT)
CaPT simplifies the prompt architecture by using exactly one "capsule" vector per Transformer layer. This capsule is a hybrid construction consisting of two primary components:
- Task-Aware Component: A single learnable vector that captures the general requirements of the downstream task.
- Instance-Aware Component: A dynamic representation derived from the current input sequence (typically the mean-pooled hidden states of the input or the previous layer).
By fusing these components (primarily through addition), the prompt becomes dynamic, adapting to every specific sentence the model processes. This single vector is then either prepended to the input sequence or added directly to the hidden states. This design ensures that the prompt is always relevant to the current context, effectively anchoring the model's attention throughout the deep layers of the network.
Experimental Setup
We evaluated CaPT across a diverse range of Natural Language Understanding (NLU) tasks using the SuperGLUE benchmark, including:
- Question Answering: BoolQ and MultiRC.
- Natural Language Inference: CB and RTE.
- Reasoning and Disambiguation: COPA and WiC.
The method was tested on several prominent architectures, including T5 (Base and Large), Llama-3.2-1B, and Qwen-2.5-1.5B, to ensure the findings generalized across both encoder-decoder and decoder-only models.
Key Results
Performance and Efficiency
CaPT consistently outperformed traditional prompt tuning and matched or exceeded full fine-tuning results. On T5-Large, CaPT achieved an average accuracy of 84.03%, compared to 78.16% for standard prompt tuning and 83.60% for full fine-tuning.
In terms of parameter efficiency, the gains were even more pronounced. For the Llama-3.2-1B model, CaPT required tuning only 0.003% of the total parameters. Because the method uses a fixed length of one vector, it completely bypasses the need for prompt-length grid searches, resulting in a 8.7x speedup in training time compared to traditional methods.
The Power of Instance Awareness
Our experiments confirmed that LLMs are "hungry" for instance-specific guidance. Even without additional training, prepending a mean-pooled version of the input to a pre-trained model improved performance. CaPT leverages this hunger by formalizing the instance-aware component within the learnable framework, ensuring the "Attention Anchor" remains robust across all layers.
Conclusion
Capsule Prompt-Tuning (CaPT) demonstrates that in the realm of prompt engineering, quality and relevance outweigh quantity. By replacing a sequence of static vectors with a single, instance-aware capsule, we can achieve higher accuracy with significantly lower computational overhead. CaPT not only solves the practical issue of prompt-length optimization but also provides a more theoretically sound way for prompts to interact with input data. Future work will explore the application of the "Attention Anchor" effect in multi-modal settings and long-form generative tasks.
Summary updated