TL;DR
This paper introduces M²PT, a novel parameter-efficient finetuning method for Multimodal Large Language Models (MLLMs). M²PT effectively integrates visual and textual prompts into both the vision encoder and language model, enabling better cross-modal understanding and significantly improving zero-shot instruction learning performance with minimal trainable parameters.
What Problem Does This Paper Address?
The rapid growth of Multimodal Large Language Models (MLLMs) has led to impressive capabilities across various domains. However, adapting these massive models to new, unseen multimodal tasks (zero-shot generalization) efficiently is a major challenge. Traditional instruction tuning, while effective, requires updating a large number of parameters, making it computationally expensive. Existing parameter-efficient finetuning (PEFT) methods often focus on single modalities, neglecting the crucial interplay between vision and language. This paper addresses the need for a parameter-efficient method that can effectively leverage multimodal information for improved zero-shot instruction learning in MLLMs.
Methods
The core innovation is the Multimodal Prompt Tuning (M²PT) approach. Instead of finetuning the entire model, M²PT introduces learnable "soft prompts" that are prepended to the input sequences of both the vision encoder and the language model.
- Visual Prompts: These are integrated into the vision encoder, helping it process visual information more effectively.
- Textual Prompts: These are incorporated into the language model to better capture textual patterns and nuances.
- Cross-modality Interaction Layer: A key component is a tunable interaction layer that explicitly aligns the features extracted by the vision encoder with the textual embeddings. This layer facilitates the synergy between visual and textual information.
The authors keep the backbone parameters of the vision encoder and LLM frozen, only training these prompt components and the interaction layer. This significantly reduces the number of trainable parameters.
Datasets
- Training: Vision-Flan, a large-scale, human-annotated multimodal instruction tuning dataset with 191 diverse tasks. For computational efficiency, a scaled-down version with 191,105 instances was used.
- Evaluation:
- MME: A comprehensive benchmark for multimodal evaluation, covering perception and cognition across 14 subtasks.
- Other Multimodal Datasets: Text-VQA (OCR), VSR (Reasoning), CIFAR-10/100 (Perception), MNIST (Perception), SNLI-VE (Visual Entailment), and POPE (Object Hallucination).
Key Insights and Ideas
- Synergy of Modalities: The paper strongly suggests that explicitly integrating and aligning prompts from both visual and textual modalities is crucial for effective zero-shot multimodal learning, outperforming methods that focus on a single modality.
- Parameter Efficiency is Key: As MLLMs grow, parameter-efficient methods are not just desirable but essential for practical deployment and adaptation. M²PT demonstrates that significant performance gains can be achieved with a tiny fraction of trainable parameters (as low as 0.09%).
- Prompt Location Matters: Experiments show that prompt depth (insertion into earlier layers) generally leads to better performance, indicating the importance of early feature extraction and alignment.
- Task-Specific Prompt Lengths: There isn't a universal "best" prompt length; optimal lengths vary depending on the task, suggesting that different tasks require different levels of detail or focus from the prompts.
Key Results
- M²PT outperforms several state-of-the-art PEFT methods (LoRA, APrompt, PTUM, VPT) on various multimodal evaluation datasets.
- M²PT achieves 94.75% of the performance of full finetuning (LLaVA-FT) while training only 0.09% of the parameters.
- In some cases, M²PT even surpasses full finetuning on specific tasks like VSR, MNIST, and POPE.
- Ablation studies confirm the effectiveness of both visual and textual prompts, as well as the cross-modality interaction layer. Removing any of these components leads to performance degradation.
- The model shows excellent performance even with significant data reduction, highlighting its efficiency and robustness.
Limitations and Future Work
- Hyperparameter Tuning: M²PT requires tuning two prompt length hyperparameters (visual and textual). While the authors found a narrow range that works well, they suggest exploring automated methods for finding optimal lengths.
- Interpretability: The paper acknowledges a lack of ad-hoc explainability for the learned prompts. While activation maps show their influence, further research is needed to understand the underlying nature of these prompts, which is critical for decision-critical applications.
- Future Directions: The authors suggest exploring more advanced network architectures for prompt length optimization and developing better interpretability methods for prompts. They also highlight the potential risks of high energy consumption associated with LLM tuning.
Summary updated