Towards Robust Multimodal Prompting with Missing Modalities

Jaehyuk Jang, Yooseung Wang, Changick Kim

TL;DR

Researchers from KAIST have developed a more efficient and robust way to handle missing data in multimodal AI (like models that process both images and text). By switching from "missing-aware" prompts (which require a unique prompt for every possible combination of missing data) to "modality-specific" prompts that are summed together, their method significantly reduces complexity while improving performance in real-world scenarios where certain data types might be unavailable during inference.

What Problem Does This Paper Address?

In an ideal world, multimodal models always have access to all data streams (e.g., a video, its transcript, and its sensor data). In reality, sensors fail, text is missing, or data is corrupted.

Previous state-of-the-art methods used Missing-Aware Prompts (MAPs). If you have $M$ modalities, MAPs require $2^M - 1$ different prompts to cover every possible combination of "present" and "missing" data. This creates two massive headaches:

  1. Exponential Scaling: As you add more modalities (depth, thermal, audio), the number of prompts explodes.
  2. Brittleness: If the model never saw a specific "missing" combination during training (e.g., it saw "image-only" and "complete" but never "text-only"), it fails completely during inference.

Methods

The authors propose Modality-Specific Prompts (MSPs). Instead of creating a prompt for every scenario, they create exactly one prompt per modality (e.g., $P_{is}$ for images and $P_{ts}$ for text).

  • Prompt Selection & Summation: If only the image is present, the model uses $P_{is}$. If only text is present, it uses $P_{ts}$. If both are present, it uses the element-wise sum ($P_{is} + P_{ts}$).
  • Frozen Backbone: The core model (a Vision-Language Transformer or ViLT) remains frozen. Only the small prompt tokens and the final classification layers are trained, making it highly parameter-efficient (tuning <1% of total parameters).
  • Orthogonality Loss ($L_{ortho}$): To prevent the prompts from learning redundant information, the authors introduce a loss function that forces the image prompt and text prompt to be mathematically "orthogonal" (dissimilar). This ensures each prompt captures the unique essence of its respective modality.

Datasets

The researchers evaluated their approach on two major multimodal benchmarks:

  1. MM-IMDb: A dataset for movie genre classification using movie posters (images) and plot summaries (text).
  2. UPMC Food-101: A dataset for food category classification using images and recipes.

In their experiments, they simulated a "missing rate" of 70%, testing various lopsided training scenarios to see how well the models handled "unseen" missing data combinations.

Key Insights and Ideas

  • Compositionality over Categorization: The "aha moment" here is that we don't need to teach a model what "missing text" looks like. We just need to teach it what "image-ness" looks like. By summing modality-specific prompts, the model naturally learns to handle the "complete" case as a composition of its parts.
  • Frequency of Updates: In the old MAPs approach, the "complete-case" prompt only gets updated when both modalities are present. In the MSP approach, the image prompt $P_{is}$ gets updated during both image-only and complete-case samples. This leads to much more frequent and stable learning for each prompt.
  • Diversity via Orthogonality: Simply having two prompts isn't enough; they might both try to learn the same easy features. Forcing orthogonality ensures the text prompt focuses on linguistic nuances while the image prompt focuses on visual features.

Key Results

  • Superior Robustness: In scenarios where the training data was unbalanced (e.g., 100% images but only 30% text), the proposed MSPs outperformed previous methods by a wide margin. On MM-IMDb, MSPs achieved an F1-Macro score of 34.11 in an unseen case where the previous SOTA (MAPs) plummeted to 20.17.
  • Efficiency: MSPs achieved these results while using significantly fewer prompts than MAPs, especially as the number of modalities increases.
  • Ablation Success: The authors proved that the Orthogonality Loss wasn't just fluff—adding $L_{ortho}$ consistently boosted performance across almost all tested scenarios by ensuring diverse representation.

Limitations and Future Work

While the paper focuses on the common image-text pair, the authors acknowledge that real-world AI is moving toward "any-to-any" modality.

  • Scaling to $M > 2$: While the theory suggests this scales linearly ($M$ prompts for $M$ modalities), the paper primarily demonstrates results on two modalities. Testing this on 4 or 5 modalities (e.g., adding audio, depth, and IMU data) remains a future step.
  • Task Variety: The current work focuses on classification. How this prompt-summation strategy affects generative tasks (like multimodal captioning) is still an open question.

The authors suggest that this "modality-case-agnostic" approach will be a foundational principle for fine-tuning large-scale models for the messy, incomplete data environments of the real world.

Summary updated

Abstract

Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.
BibTeX
@inproceedings{jang2024towards,
  author = {Jaehyuk Jang and Yooseung Wang and Changick Kim},
  title = {{Towards Robust Multimodal Prompting with Missing Modalities}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=CMS32jjkRs},
}

Loading related content...

Figure