MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality

Ruiting Dai, Yuqiao Tan, Lisi Mo, Tao He, Ke Qin, Shuang LiangarXiv 2024 · OpenReview

Abstract

Recently, prompt learning has garnered considerable attention for its success in various Vision-Language (VL) tasks. However, existing prompt-based models are primarily focused on studying prompt generation and prompt strategies with complete modality settings, which does not accurately reflect real-world scenarios where partial modality information may be missing. In this paper, we present the first comprehensive investigation into prompt learning behavior when modalities are incomplete, revealing the high sensitivity of prompt-based models to missing modalities. To this end, we propose a novel Multi-step Adaptive Prompt Learning (MuAP) framework, aiming to generate multimodal prompts and perform multi-step prompt tuning, which adaptively learns knowledge by iteratively aligning modalities. Specifically, we generate multimodal prompts for each modality and devise prompt strategies to integrate them into the Transformer model. Subsequently, we sequentially perform prompt tuning from single-stage and alignment-stage, allowing each modality-prompt to be autonomously and adaptively learned, thereby mitigating the imbalance issue caused by only textual prompts that are learnable in previous works. Extensive experiments demonstrate the effectiveness of our MuAP and this model achieves significant improvements compared to the state-of-the-art on all benchmark datasets.

Introduction

Vision-Language (VL) models like CLIP have revolutionized multimodal understanding by learning shared representations between images and text. However, these models typically assume ideal conditions where both modalities are present. In real-world scenarios, data is often incomplete due to sensor failures, privacy constraints, or transmission errors.

Existing prompt-learning methods are highly sensitive to these gaps. Previous attempts to address missing modalities, such as Missing-aware Prompts (MPVR), required unique prompts for every possible combination of missing data. As the number of modalities ($C$) increases, this leads to an exponential explosion ($2^C-1$) of parameters, making the system difficult to scale.

Comparison of prompt learning paradigms: (a) standard CLIP, (b) exponential missing-aware prompts, and (c) the proposed linear Multi-view Adaptive Prompt (MuAP).

Methodology

To overcome the limitations of prior work, we introduce the MuAP framework, which maintains parameter efficiency while improving robustness through three core modules.

1. Multimodal Prompt Generator

Instead of pre-defining prompts for every missing data scenario, MuAP utilizes a lightweight network ($f_{missing}$) to "imagine" missing information. When a modality is absent, the generator uses the available modality's prompt to derive a "missing-type" prompt. This ensures that the number of prompts grows linearly with the number of modalities, rather than exponentially.

2. Prompt Strategy Design

We explore two primary methods for injecting these adaptive prompts into the Transformer backbone (e.g., ViLT):

  • Head-fusion: Summing multimodal prompts at the beginning of the sequence.
  • Cross-fusion: Inserting modality-specific prompts directly into their respective input streams.
The MuAP architecture, illustrating the Multimodal Prompt Generator, Prompt Strategy Design (Cross and Head Fusion), and the Multi-step Prompt Tuning process.

3. Multi-step Prompt Tuning

A critical innovation of MuAP is its sequential tuning process, designed to prevent the model from over-relying on a single modality (often text). The process follows three stages:

  • Text-step: Optimize text prompts while keeping image prompts frozen.
  • Image-step: Optimize image prompts while keeping text prompts frozen.
  • Alignment-stage: Jointly fine-tune all components using a classification loss and KL divergence to ensure modality distributions remain aligned.

Experiments and Results

MuAP was evaluated on three diverse benchmarks: MM-IMDb (movie genre classification), UPMC Food-101 (noisy web data), and Hateful Memes (complex image-text interplay).

Performance and Robustness

MuAP consistently outperformed state-of-the-art baselines, particularly as the rate of missing data increased. On the challenging Hateful Memes dataset, MuAP (Head Fusion) achieved a 65.09 AUROC at a 70% missing rate, significantly higher than the baseline ViLT (60.83) and MPVR (61.01). In "missing-text" scenarios, MuAP demonstrated remarkable stability, with a performance drop of only ~3.05%, compared to nearly 10% for competing methods.

Ablation Insights

Our analysis of prompt length indicates a "Goldilocks zone" for performance. We found that a length of 12–16 tokens is optimal; shorter prompts provide insufficient context, while longer prompts tend to overfit and interfere with the original input sequence's integrity.

Conclusion

MuAP addresses the critical challenge of missing modalities in Vision-Language models through a parameter-efficient, generative approach. By utilizing a multi-step tuning strategy and a linear prompt generation mechanism, it achieves superior robustness and scalability compared to existing methods. Future research will focus on extending these techniques to larger-scale models and a wider variety of modalities, such as audio and depth data.

Summary updated

Abstract

Recently, prompt learning has garnered considerable attention for its success in various Vision-Language (VL) tasks. However, existing prompt-based models are primarily focused on studying prompt generation and prompt strategies with complete modality settings, which does not accurately reflect real-world scenarios where partial modality information may be missing. In this paper, we present the first comprehensive investigation into prompt learning behavior when modalities are incomplete, revealing the high sensitivity of prompt-based models to missing modalities. To this end, we propose a novel Multi-step Adaptive Prompt Learning (MuAP) framework, aiming to generate multimodal prompts and perform multi-step prompt tuning, which adaptively learns knowledge by iteratively aligning modalities. Specifically, we generate multimodal prompts for each modality and devise prompt strategies to integrate them into the Transformer model. Subsequently, we sequentially perform prompt tuning from single-stage and alignment-stage, allowing each modality-prompt to be autonomously and adaptively learned, thereby mitigating the imbalance issue caused by only textual prompts that are learnable in previous works. Extensive experiments demonstrate the effectiveness of our MuAP and this model achieves significant improvements compared to the state-of-the-art on all benchmark datasets
BibTeX
@misc{dai2024muap,
  author = {Ruiting Dai and Yuqiao Tan and Lisi Mo and Tao He and Ke Qin and Shuang Liang},
  title = {{MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality}},
  year = {2024},
  eprint = {2409.04693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04693},
}
Figure