Prompt Tuning for Unified Multimodal Pretrained Models

Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang ZhouACL 2023 · OpenReview

Presented at ACL 2023, this research explores efficient methods for adapting massive multimodal models to specific downstream tasks without the prohibitive computational costs associated with traditional full-parameter finetuning.

Introduction

In the current landscape of artificial intelligence, large-scale pretrained models like OFA (One-For-All) have demonstrated remarkable capabilities in handling vision and language tasks simultaneously. However, the standard adaptation method—full finetuning—requires updating every parameter in the model for each new task. For models containing hundreds of millions or billions of parameters, this process is extremely resource-intensive in terms of both storage and computation.

While the Natural Language Processing (NLP) community has successfully adopted "prompt tuning" (optimizing a small set of input embeddings), its effectiveness in unified multimodal architectures—which handle understanding and generation in a single sequence-to-sequence framework—remained largely unexplored until now. The authors introduce OFA-PT, a parameter-efficient method that tunes less than 1% of the model's parameters while achieving performance parity with full finetuning.

Methodology: OFA-PT Architecture

The researchers implemented a specific variation of prompt tuning known as prefix tuning on the OFA model. Unlike standard prompt tuning which only modifies input embeddings, prefix tuning inserts learnable parameters into the hidden states of the model.

Figure 1: The OFA-PT architecture inserts learnable prompt embeddings into every layer of both the encoder and decoder while keeping the pretrained backbone frozen.

Key aspects of the OFA-PT approach include:

  1. Frozen Backbone: The entire pretrained Transformer backbone remains frozen during training.
  2. Soft Prompts: A sequence of learnable embeddings (soft prompts) is inserted into every layer of both the encoder and the decoder.
  3. Parameter Efficiency: This setup utilizes only about 1% of the total parameters compared to full finetuning.
  4. Configuration: Through "hyperparameter surgery," the authors determined that an "EncDec" setup (prompts in both components) with a prompt length of 64 tokens provided the optimal balance of performance and efficiency.

Robustness to Adversarial Attacks

One of the most significant findings of the study is that prompt-tuned models exhibit superior robustness compared to their fully finetuned counterparts. When subjected to adversarial attacks (such as the Fast Gradient Sign Method, or FGSM), the performance of finetuned models degrades significantly more than that of prompt-tuned models.

Figure 4: Comparison of performance reduction under adversarial attacks across multiple datasets. Prompt tuning (light blue) consistently shows lower performance degradation than full finetuning (dark blue).

Because the core weights of the pretrained model are frozen in OFA-PT, the model is less likely to overfit to the specific noise or adversarial patterns introduced during the attack. This suggests that the frozen weights act as a form of "armor," preserving the generalizable knowledge captured during pretraining.

Key Insights and Results

The Scale Threshold

The success of prompt tuning is heavily dependent on model size. On the "Base" model (~180M parameters), prompt tuning lagged behind finetuning. However, on the "Large" model (~470M parameters), the performance gap virtually disappeared. This indicates that larger models possess more latent knowledge that can be effectively unlocked with a minimal number of trainable parameters.

Performance and Efficiency

  • Parity: On the OFA-Large model, OFA-PT achieved nearly identical results to full finetuning. For instance, on the RefCOCO (val) benchmark, both methods achieved an accuracy of 90.05%.
  • Competitive Edge: OFA-PT consistently outperformed other popular parameter-efficient methods like Adapters and BitFit across various benchmarks.
  • Training Speed: For the "Huge" model variant, prompt tuning was approximately 2x faster per 100 samples during training compared to traditional finetuning.

Limitations and Future Work

Despite its advantages, prompt tuning presents several challenges. It is notably more sensitive to hyperparameters (such as learning rates) than standard finetuning. Additionally, while individual training steps are faster, prompt-tuned models often require more epochs to reach peak performance. Finally, the "small model gap" remains; the method currently provides the most benefit to those utilizing the largest available architectures.

Future research will likely focus on stabilizing the training process and developing automated methods for finding optimal prompt configurations, potentially making parameter-efficient tuning more accessible for smaller-scale models.

Summary updated

BibTeX
@inproceedings{yang2023prompt,
  author = {Hao Yang and Junyang Lin and An Yang and Peng Wang and Chang Zhou},
  title = {{Prompt Tuning for Unified Multimodal Pretrained Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=4e51O_T94TJ},
}
Figure