Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement

Wuliang Huang, Yiqiang Chen, Xinlong Jiang, Chenlong Gao, Teng Zhang, Qian Chen, Yifan Wang

Abstract

The Multimodal Generalization and Refinement (MGR) framework addresses the challenge of "pervasive modality absence," where data streams like audio or text are missing during both training and inference. Unlike traditional methods that rely on imputation or simple joint learning, MGR introduces a "recycle, don't trash" philosophy. It utilizes mixed-context prompting to identify biases and graph-based redistribution to preserve useful information from biased tokens. Evaluated on benchmarks like CMU-MOSEI, MGR demonstrates superior robustness and performance, even in extreme scenarios where all training and testing samples contain missing modalities.

Introduction

Most multimodal AI models assume that all data streams (vision, audio, text) are consistently available. However, real-world applications often face "pervasive modality absence" due to sensor failures, environmental noise, or privacy restrictions. When a model never encounters "complete" data during training, it learns biased correlations that lead to significant distribution shifts during deployment.

t-SNE visualizations of feature distributions in baseline models (Transformer, MAPs, MSPs), showing distinct clustering based on missing modality patterns rather than semantic content.

Existing solutions typically attempt to hallucinate missing data (imputation) or learn robust features from available streams. Imputation often requires complete datasets for supervision, which are unavailable in pervasive scenarios. Meanwhile, joint learning methods frequently fail to mitigate the biases inherent in incomplete data streams, as seen in the clear separation of feature clusters based on absence patterns in baseline models.

The MGR Framework

The authors propose the Multimodal Generalization and Refinement (MGR) framework to tackle these challenges through a two-stage process: representation generalization and iterative refinement.

The overall architecture of the MGR framework, illustrating the Multimodal Representation Generalization stage and the iterative Potential Biases Identification and Refinement process.

1. Multimodal Representation Generalization

MGR leverages a multimodal foundation model (e.g., ViLT) to generate initial embeddings. To enhance robustness, it integrates independent unimodal features from separate Transformers. Since unimodal models are agnostic to missing data in other streams, their features provide a stable "anchor" for the multimodal representations. A diversity loss ($\mathcal{L}_{div}$) is employed to ensure that multimodal and unimodal representations remain complementary rather than redundant.

2. Potential Biases Identification and Refinement

The core innovation of MGR is its iterative refinement process, which moves beyond simple fusion:

  • Mixed-Context Prompting: The model generates prompts combining the "absence status" (what is missing) with the "global context" (the actual content), allowing for precise localization of biases.
  • Bias Identification: Tokens are ranked based on feature reliability and their correlation with other tokens in the sequence.
  • Graph-based Redistribution: Instead of discarding biased tokens, MGR treats them as nodes in a graph. It uses graph pooling to redistribute reliable information from biased tokens into unbiased ones, effectively "recycling" subtle cues that would otherwise be lost if the tokens were simply deleted.

Experimental Results

MGR was evaluated on four major benchmarks: CMU-MOSI, CMU-MOSEI, UR-FUNNY, and AV-MNIST. The framework was tested under varying absence prevalence ratios ($\kappa$), including the extreme case of $\kappa=1.0$, where every single sample in the dataset is incomplete.

  • Superior Performance: On the CMU-MOSEI dataset with a 70% absence rate, MGR achieved an F1 score of 85.27, significantly higher than the baseline's 82.20.
  • Extreme Robustness: MGR maintained high accuracy (70.26% average across three datasets) even when all training and testing samples had missing modalities, while other methods saw sharper performance drops.
  • Efficiency: The iterative refinement process (using $L_T$ layers) was shown to converge quickly, providing robustness without a massive computational cost.

Conclusion

The MGR framework shifts the paradigm from discarding biased data to "recycling" it. By anchoring multimodal representations with unimodal features and using graph-based redistribution, MGR achieves state-of-the-art performance in pervasive modality absence scenarios. Future work will explore scaling this approach to a wider variety of modalities and integrating it with larger multimodal foundation models.

Summary updated

Abstract

The performance of multimodal models often deteriorates when modality absence occurs. The absence disrupts the learned inter-modal correlations, resulting in biased multimodal representations. This challenge is especially pronounced when the absence is pervasive, affecting both the training and inference phases. Recent studies have attempted to reconstruct the missing information; however, most of them require complete supervision, which is seldom available in scenarios of pervasive absence. The quality of reconstruction remains a critical issue. Alternatively, others aim to learn robust representations from the available modalities but the substantial variations and biases are not fully addressed. This paper introduces the Multimodal Generalization and Refinement (MGR) framework to mitigate the issue of pervasive modality absence. MGR begins by acquiring generalized multimodal representations and iteratively refines them to recognize and calibrate the biased representations. Initially, multimodal samples with absence are embedded through foundation models, and MGR integrates independent unimodal features to further enhance generalization. Additionally, a novel mixed-context prompt is adopted to identify biases in both features and correlations. A redistribution operation can then refine these biases through graph pooling, culminating in robust and calibrated multimodal representations, which are suitable for downstream tasks. Comprehensive experiments on four benchmark datasets demonstrate that the proposed MGR framework outperforms state-of-the-art methods, effectively mitigating the impact of pervasive modality absence.
BibTeX
@inproceedings{huang2025mitigating,
  author = {Wuliang Huang and Yiqiang Chen and Xinlong Jiang and Chenlong Gao and Teng Zhang and Qian Chen and Yifan Wang},
  title = {{Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i25.34883},
}
Figure