Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations

Junsong Chen, Jiyuan Liu, Suyuan Liu, Wei Emma Zhang, Ao Li, En Zhu, Xinwang LiuAAAI 2026 · DOI

Abstract

In multimodal sentiment analysis, modality missingness and quality degradation are common. Existing methods often rely on batch-level modality generation, generation but neglect sample-level missingness, hence their flexibility is limited severely in real-world scenarios. To address this, Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (SMCIR) is proposed. Specifically, The Dynamic Multi-feature Fusion Detector (DMFD) is presented, which detects missingness and severity at the sample-level using indicators such as information entropy, modality similarity, and mutual information. Unlike batch-based methods, the DMFD provides fine-grained detection and adaptive responses, improving sensitivity to modality disturbances. Meanwhile, the Context-aware Modality Completion Generator (CMCG) is developed to restore missing modalities through context-guided reconstruction using multiscale feature fusion and cross-modal attention. In this way, the proposed CMCG method can avoid redundancy and inconsistency, enhancing the consistency and discriminativity of the fused representation. In CMCG, the text modality serves as a stable guide to improve context consistency. Experiments on the CMU-MOSI and CMU-MOSEI datasets show that SMCIR outperforms existing full-modal and non-recovery-based methods, well validating its efficacy and superiority in multimodal learning.
BibTeX
@inproceedings{chen2026sample,
  author = {Junsong Chen and Jiyuan Liu and Suyuan Liu and Wei Emma Zhang and Ao Li and En Zhu and Xinwang Liu},
  title = {{Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i24.39102},
}

Loading related content...

Figure