Abstract
Prompt Tuning has been largely successful as a parameter-efficient method of conditioning large-scale pre-trained language models to perform downstream tasks. Thus far, soft prompt tuning learns a fixed set of task-specific continuous vectors, i.e., soft tokens that remain static across the task samples. A fixed prompt, however, may not generalize well to the diverse kinds of inputs the task comprises. In order to address this, we propose Vector-quantized Input-contextualized Prompts (VIP) as an extension to the soft prompt tuning framework. VIP particularly focuses on two aspects -- contextual prompts that learns input-specific contextualization of the soft prompt tokens through a small-scale sentence encoder and quantized prompts that maps the contextualized prompts to a set of learnable codebook vectors through a Vector quantization network. On various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI, VIP outperforms the soft prompt tuning (PT) baseline by an average margin of 1.19%. Further, our generalization studies show that VIP learns more robust prompt representations, surpassing PT by a margin of 0.6% - 5.3% on Out-of-domain QA and NLI tasks respectively, and by 0.75% on Multi-Task setup over 4 tasks spanning across 12 domains.
Introduction
Fine-tuning massive pre-trained language models (PLMs) for every new task is computationally expensive. Prompt Tuning (PT) emerged as a "green" alternative: instead of changing the model's billions of weights, only a small set of "soft prompt" vectors prepended to the input are learned. However, standard prompt tuning uses static prompts that remain identical regardless of the input. This lack of flexibility hinders performance on diverse inputs or out-of-domain data. Previous attempts to make prompts dynamic often suffered from training instability and "representation collapse."
Methodology: Vector-quantized Input-contextualized Prompts (VIP)
The VIP framework introduces two key components to the standard prompt tuning pipeline to enable dynamic, input-aware prompting while maintaining stability.
- Prompt Contextualization: VIP employs a tiny, trainable Transformer-based sentence encoder (~86K parameters). This encoder processes both the input text ($x$) and the static prompt ($p_i$) to generate input-adapted prompt tokens ($p_i^c$).
- Prompt Quantization: To prevent noise from the contextualizer from destabilizing the frozen PLM, VIP passes the contextualized tokens through a Vector Quantization (VQ) layer. This layer maps continuous vectors to a discrete "codebook" of learnable vectors ($p_i^q$).
- Hybrid Prompting: The final prompt is a combination of the original static prompt and the new quantized, contextualized prompt ($p_i + p_i^q$). This skip-connection ensures the model retains stable task-specific information while gaining input-specific nuances.
Experimental Results
VIP was evaluated across a wide range of NLU benchmarks, including SuperGLUE, MRQA, and Adversarial NLI. The method consistently outperforms standard Prompt Tuning (PT) across all benchmarks.
Performance and Efficiency
VIP outperformed the PT baseline by an average margin of 1.19%. As shown in the figure above, while VIP uses slightly more parameters than basic PT (0.9M vs 0.1M), it remains significantly more efficient than full fine-tuning (FT) or Adapter-based methods while achieving competitive scores on SuperGLUE.
Robustness and Generalization
The gains were most pronounced in out-of-domain (OOD) scenarios. VIP surpassed PT by 0.6% on OOD Question Answering and 5.3% on OOD Natural Language Inference. By learning a codebook of "prompt-atoms," the model can map unfamiliar OOD inputs to relevant discrete concepts, leading to better generalization.
The Role of Quantization
A critical finding is that "raw" contextualized prompts (without quantization) often perform worse than static prompts. Quantization acts as a vital regularizer, filtering out noise and preventing "posterior collapse" where the model ignores the input context.
Conclusion
VIP demonstrates that dynamic, input-contextualized prompts can significantly enhance NLU performance and robustness without sacrificing the parameter efficiency of prompt tuning. Future work includes exploring task-agnostic pre-training for the VIP contextualizer and scaling the approach to even larger language model backbones.
Summary updated