Prompt Tuning for Generative Multimodal Pretrained Models

Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou, Hongxia YangarXiv 2022 · OpenReview

Abstract

Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning.

Introduction

Large-scale pretrained models have revolutionized AI, but the standard approach of full fine-tuning is computationally and storage-intensive, requiring the update of billions of parameters for every downstream task. While "prompt tuning"—learning a small set of "soft" prompt embeddings while keeping the main model frozen—has succeeded in Natural Language Processing (NLP), its application to generative multimodal models remained largely unexplored. Most previous multimodal prompting research focused on contrastive models like CLIP, which lack the flexibility for complex generative tasks like image captioning or visual question answering.

Methodology

The researchers implemented prefix tuning on a state-of-the-art unified sequence-to-sequence (Seq2Seq) model called OFA (One-For-All). Unlike manual "hard" prompts, this approach uses learnable "soft" prompt embeddings.

Architecture of prompt tuning in the OFA encoder-decoder framework.

A key technical innovation is Deep Prompting, where tunable embeddings are inserted into every layer of both the encoder and the decoder (as shown in Figure 1). This allows the model to better integrate the prompt information throughout the understanding and generation processes. The study systematically evaluated the impact of prompt length, prompt depth (encoder vs. decoder), and reparameterization using MLPs for embedding generation.

Experimental Results

The approach was evaluated across diverse tasks including Referring Expression Comprehension (RefCOCO/+/g), Visual Entailment (SNLI-VE), Image Captioning (MS COCO), and Visual Question Answering (VQA 2.0).

Parameter Efficiency and Performance

Prompt tuning achieved comparable results to full fine-tuning while updating only about 1% of the parameters. On the RefCOCO task using a "Large" model, prompt tuning achieved a validation score of 90.05, matching full fine-tuning. Interestingly, a scaling effect was observed: while prompt tuning lags on "Base" models, the performance gap disappears as model size increases, suggesting larger models possess more accessible latent knowledge.

Impact of Prompt Length

The study found that performance is sensitive to prompt length. While increasing length generally helps, there is a "sweet spot"—typically around 64 tokens—beyond which returns diminish or performance degrades.

Performance vs. Prompt Length on the RefCOCO dataset.

Adversarial Robustness

By keeping pretrained weights frozen, prompt-tuned models retain general-purpose knowledge and avoid the "overfitting" often seen in fine-tuning. This results in significantly higher resilience to adversarial noise (e.g., FGSM attacks). For instance, on RefCOCO+, the performance drop under attack for fine-tuned models was nearly double that of prompt-tuned ones.

Discussion and Limitations

Despite its efficiency, prompt tuning faces challenges. It typically requires more training epochs (40+) to converge compared to fine-tuning. It is also highly sensitive to initial hyperparameters like prompt length and depth. Furthermore, a performance gap remains for smaller model architectures. Future research should focus on better initialization strategies to speed up convergence and further explore the theoretical foundations of the observed adversarial robustness.

Summary updated

Abstract

Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning. Despite the observed advantages, we still find some limitations in prompt tuning, and we correspondingly point out the directions for future studies. Codes are available at \url{https://github.com/OFA-Sys/OFA}
BibTeX
@misc{yang2022prompt,
  author = {Hao Yang and Junyang Lin and An Yang and Peng Wang and Chang Zhou and Hongxia Yang},
  title = {{Prompt Tuning for Generative Multimodal Pretrained Models}},
  year = {2022},
  eprint = {2208.02532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.02532},
}
Figure