Abstract
Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning.
Introduction
Large-scale pretrained models have revolutionized AI, but the standard approach of full fine-tuning is computationally and storage-intensive, requiring the update of billions of parameters for every downstream task. While "prompt tuning"—learning a small set of "soft" prompt embeddings while keeping the main model frozen—has succeeded in Natural Language Processing (NLP), its application to generative multimodal models remained largely unexplored. Most previous multimodal prompting research focused on contrastive models like CLIP, which lack the flexibility for complex generative tasks like image captioning or visual question answering.
Methodology
The researchers implemented prefix tuning on a state-of-the-art unified sequence-to-sequence (Seq2Seq) model called OFA (One-For-All). Unlike manual "hard" prompts, this approach uses learnable "soft" prompt embeddings.
A key technical innovation is Deep Prompting, where tunable embeddings are inserted into every layer of both the encoder and the decoder (as shown in Figure 1). This allows the model to better integrate the prompt information throughout the understanding and generation processes. The study systematically evaluated the impact of prompt length, prompt depth (encoder vs. decoder), and reparameterization using MLPs for embedding generation.
Experimental Results
The approach was evaluated across diverse tasks including Referring Expression Comprehension (RefCOCO/+/g), Visual Entailment (SNLI-VE), Image Captioning (MS COCO), and Visual Question Answering (VQA 2.0).
Parameter Efficiency and Performance
Prompt tuning achieved comparable results to full fine-tuning while updating only about 1% of the parameters. On the RefCOCO task using a "Large" model, prompt tuning achieved a validation score of 90.05, matching full fine-tuning. Interestingly, a scaling effect was observed: while prompt tuning lags on "Base" models, the performance gap disappears as model size increases, suggesting larger models possess more accessible latent knowledge.
Impact of Prompt Length
The study found that performance is sensitive to prompt length. While increasing length generally helps, there is a "sweet spot"—typically around 64 tokens—beyond which returns diminish or performance degrades.
Adversarial Robustness
By keeping pretrained weights frozen, prompt-tuned models retain general-purpose knowledge and avoid the "overfitting" often seen in fine-tuning. This results in significantly higher resilience to adversarial noise (e.g., FGSM attacks). For instance, on RefCOCO+, the performance drop under attack for fine-tuned models was nearly double that of prompt-tuned ones.
Discussion and Limitations
Despite its efficiency, prompt tuning faces challenges. It typically requires more training epochs (40+) to converge compared to fine-tuning. It is also highly sensitive to initial hyperparameters like prompt length and depth. Furthermore, a performance gap remains for smaller model architectures. Future research should focus on better initialization strategies to speed up convergence and further explore the theoretical foundations of the observed adversarial robustness.
Summary updated