Abstract
Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.
Introduction
In real-world applications, multimodal models often face missing data due to sensor failure, data corruption, or incomplete records. Previous state-of-the-art methods addressed this using Missing-Aware Prompts (MAPs). However, MAPs suffer from two major drawbacks. First, they require $2^M - 1$ different prompts for $M$ modalities, leading to exponential scaling as more data types (e.g., audio, thermal) are added. Second, they are brittle; if a specific combination of missing data was not encountered during training, the model often fails during inference.
Methods: Modality-Specific Prompts
To overcome these limitations, we propose Modality-Specific Prompts (MSPs). Instead of creating a unique prompt for every possible missing-data scenario, we create exactly one prompt per modality (e.g., $P_{is}$ for images and $P_{ts}$ for text).
Prompt Selection and Summation
The core innovation is compositionality. If only an image is present, the model uses $P_{is}$. If only text is present, it uses $P_{ts}$. For the "complete case" where both are present, the model uses the element-wise sum ($P_{is} + P_{ts}$). This allows the model to learn the unique characteristics of each modality independently and combine them naturally, rather than treating every missing-data combination as a separate category.
Parameter Efficiency and Orthogonality
The backbone model (e.g., a Vision-Language Transformer) remains frozen, with only the small prompt tokens and final classification layers being learnable. This makes the approach highly parameter-efficient, tuning less than 1% of total parameters. To ensure that prompts capture distinct information, we introduce an Orthogonality Loss ($L_{ortho}$). This forces the image and text prompts to be mathematically dissimilar, preventing redundancy and ensuring each prompt focuses on the unique features of its respective modality.
Experimental Results
We evaluated MSPs on the MM-IMDb (movie genre classification) and UPMC Food-101 (food category classification) benchmarks. We simulated a 70% missing rate and tested various unbalanced training scenarios to evaluate robustness against "unseen" missing modality combinations.
Robustness and Efficiency
MSPs demonstrate superior robustness in scenarios where training and inference distributions differ. For instance, on MM-IMDb, when trained on 100% images but only 30% text, MSPs achieved an F1-Macro score of 34.11 in the text-heavy inference case, whereas the previous SOTA (MAPs) plummeted to 20.17. Furthermore, MSPs achieve these results with significantly fewer parameters as the number of modalities increases, maintaining linear rather than exponential growth.
Conclusion
By shifting from scenario-based categorization to modality-based composition, MSPs provide a more scalable and robust framework for multimodal learning with missing data. This "modality-case-agnostic" approach is a promising step toward deploying large-scale models in the messy, incomplete data environments of the real world. Future work will explore scaling this to more than two modalities and its application to generative tasks.
Summary updated