Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations
Lists
Only you see this
Abstract
In multimodal sentiment analysis, modality missingness and quality degradation are common. Existing methods often rely on batch-level modality generation, generation but neglect sample-level missingness, hence their flexibility is limited severely in real-world scenarios. To address this, Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (SMCIR) is proposed. Specifically, The Dynamic Multi-feature Fusion Detector (DMFD) is presented, which detects missingness and severity at the sample-level using indicators such as information entropy, modality similarity, and mutual information. Unlike batch-based methods, the DMFD provides fine-grained detection and adaptive responses, improving sensitivity to modality disturbances. Meanwhile, the Context-aware Modality Completion Generator (CMCG) is developed to restore missing modalities through context-guided reconstruction using multiscale feature fusion and cross-modal attention. In this way, the proposed CMCG method can avoid redundancy and inconsistency, enhancing the consistency and discriminativity of the fused representation. In CMCG, the text modality serves as a stable guide to improve context consistency. Experiments on the CMU-MOSI and CMU-MOSEI datasets show that SMCIR outperforms existing full-modal and non-recovery-based methods, well validating its efficacy and superiority in multimodal learning.
BibTeX
@inproceedings{chen2026sample,
author = {Junsong Chen and Jiyuan Liu and Suyuan Liu and Wei Emma Zhang and Ao Li and En Zhu and Xinwang Liu},
title = {{Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations}},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2026},
doi = {10.1609/aaai.v40i24.39102},
}Loading related content...