Abstract
The Multimodal Generalization and Refinement (MGR) framework addresses the challenge of "pervasive modality absence," where data streams like audio or text are missing during both training and inference. Unlike traditional methods that rely on imputation or simple joint learning, MGR introduces a "recycle, don't trash" philosophy. It utilizes mixed-context prompting to identify biases and graph-based redistribution to preserve useful information from biased tokens. Evaluated on benchmarks like CMU-MOSEI, MGR demonstrates superior robustness and performance, even in extreme scenarios where all training and testing samples contain missing modalities.
Introduction
Most multimodal AI models assume that all data streams (vision, audio, text) are consistently available. However, real-world applications often face "pervasive modality absence" due to sensor failures, environmental noise, or privacy restrictions. When a model never encounters "complete" data during training, it learns biased correlations that lead to significant distribution shifts during deployment.
Existing solutions typically attempt to hallucinate missing data (imputation) or learn robust features from available streams. Imputation often requires complete datasets for supervision, which are unavailable in pervasive scenarios. Meanwhile, joint learning methods frequently fail to mitigate the biases inherent in incomplete data streams, as seen in the clear separation of feature clusters based on absence patterns in baseline models.
The MGR Framework
The authors propose the Multimodal Generalization and Refinement (MGR) framework to tackle these challenges through a two-stage process: representation generalization and iterative refinement.
1. Multimodal Representation Generalization
MGR leverages a multimodal foundation model (e.g., ViLT) to generate initial embeddings. To enhance robustness, it integrates independent unimodal features from separate Transformers. Since unimodal models are agnostic to missing data in other streams, their features provide a stable "anchor" for the multimodal representations. A diversity loss ($\mathcal{L}_{div}$) is employed to ensure that multimodal and unimodal representations remain complementary rather than redundant.
2. Potential Biases Identification and Refinement
The core innovation of MGR is its iterative refinement process, which moves beyond simple fusion:
- Mixed-Context Prompting: The model generates prompts combining the "absence status" (what is missing) with the "global context" (the actual content), allowing for precise localization of biases.
- Bias Identification: Tokens are ranked based on feature reliability and their correlation with other tokens in the sequence.
- Graph-based Redistribution: Instead of discarding biased tokens, MGR treats them as nodes in a graph. It uses graph pooling to redistribute reliable information from biased tokens into unbiased ones, effectively "recycling" subtle cues that would otherwise be lost if the tokens were simply deleted.
Experimental Results
MGR was evaluated on four major benchmarks: CMU-MOSI, CMU-MOSEI, UR-FUNNY, and AV-MNIST. The framework was tested under varying absence prevalence ratios ($\kappa$), including the extreme case of $\kappa=1.0$, where every single sample in the dataset is incomplete.
- Superior Performance: On the CMU-MOSEI dataset with a 70% absence rate, MGR achieved an F1 score of 85.27, significantly higher than the baseline's 82.20.
- Extreme Robustness: MGR maintained high accuracy (70.26% average across three datasets) even when all training and testing samples had missing modalities, while other methods saw sharper performance drops.
- Efficiency: The iterative refinement process (using $L_T$ layers) was shown to converge quickly, providing robustness without a massive computational cost.
Conclusion
The MGR framework shifts the paradigm from discarding biased data to "recycling" it. By anchoring multimodal representations with unimodal features and using graph-based redistribution, MGR achieves state-of-the-art performance in pervasive modality absence scenarios. Future work will explore scaling this approach to a wider variety of modalities and integrating it with larger multimodal foundation models.
Summary updated