Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification

Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, Jianhua Yao

Introduction

In safety-critical domains like oncology, integrating diverse data sources—such as mRNA expression, DNA methylation, and miRNA—is essential for accurate diagnosis. However, multimodal data often contains noise or irrelevant information that varies across individual samples. Traditional "black-box" fusion techniques typically assume uniform importance across all modalities, leading to suboptimal performance and a lack of transparency. This paper introduces Multimodal Dynamics, a framework designed to enhance trustworthiness by dynamically assessing data informativeness at both the feature and modality levels.

Methodology: Dynamical Fusion

The core innovation lies in a two-stage "Dynamical Fusion" strategy that filters noise and weights modalities based on their sample-specific reliability.

1. Feature-level Dynamics

To handle high-dimensional noise, the framework employs a sparse gating network for each modality. This network generates a sample-specific mask that selectively activates informative features while suppressing noise. By applying $\ell_1$ regularization, the model identifies the most relevant biomarkers for each specific patient, facilitating inherent explainability.

2. Modality-level Dynamics

Recognizing that different modalities offer varying levels of certainty for different samples, the authors utilize True-Class-Probability (TCP) as a confidence metric. Since the true class is unknown during inference, a dedicated confidence regression network is trained to estimate the TCP for each modality.

Architecture of the Multimodal Dynamics framework, illustrating feature-level gating and modality-level confidence estimation.

3. Integration

The final joint representation is formed by weighting each modality's features by its estimated confidence before concatenation and classification. This ensures that the model prioritizes the most "certain" data sources for every decision.

Experimental Evaluation

The framework was validated using four multiomics datasets from The Cancer Genome Atlas (TCGA): BRCA (Breast invasive carcinoma), LGG (Brain glioma), KIPAN (Kidney cancer), and ROSMAP (Alzheimer’s Disease). Each dataset comprises three modalities: mRNA, DNA methylation, and miRNA.

Results and Discussion

The Multimodal Dynamics approach consistently outperformed state-of-the-art (SOTA) fusion methods and single-modality baselines.

Performance Gains

On the BRCA dataset, the model achieved an accuracy of 87.7%, surpassing the previous SOTA (TMC) by 3.5%. Similar trends were observed across other metrics and datasets, demonstrating the robustness of dynamical fusion.

Performance metrics (ACC, F1_weighted, F1_macro) for the BRCA dataset across different modality combinations.

Explainability and Stability

Beyond accuracy, the model provides transparency. Qualitative analysis revealed that the features prioritized by the gating network often align with known biological biomarkers (e.g., KLK8, PTX3). Furthermore, the filtering of uninformative data resulted in lower prediction variance compared to standard concatenation methods.

Conclusion

Multimodal Dynamics addresses the critical need for trustworthy AI in medicine by moving away from static fusion. By dynamically evaluating informativeness at multiple scales, the framework achieves superior performance while offering insights into the underlying biological signals. Future work will focus on optimizing computational efficiency for scenarios involving a larger number of modalities and exploring robustness against noisy labels.

Summary updated

Abstract

Integration of heterogeneous and high-dimensional data (e.g., multiomics) is becoming increasingly important. Existing multimodal classification algorithms mainly focus on improving performance by exploiting the complementarity from different modalities. However, conventional approaches are basically weak in providing trustworthy multimodal fusion, especially for safety-critical applications (e.g., medical diagnosis). For this issue, we propose a novel trustworthy multimodal classification algorithm termed Multimodal Dynamics, which dynamically evaluates both the feature-level and modality-level informativeness for different samples and thus trustworthily integrates multiple modalities. Specifically, a sparse gating is introduced to capture the information variation of each within-modality feature and the true class probability is employed to assess the classification confidence of each modality. Then a transparent fusion algorithm based on the dynamical informativeness estimation strategy is induced. To the best of our knowledge, this is the first work to jointly model both feature and modality variation for different samples to provide trustworthy fusion in multi-modal classification. Extensive experiments are conducted on multimodal medical classification datasets. In these experiments, superior performance and trustworthiness of our algorithm are clearly validated compared to the state-of-the-art methods.
BibTeX
@inproceedings{han2022multimodal,
  author = {Zongbo Han and Fan Yang and Junzhou Huang and Changqing Zhang and Jianhua Yao},
  title = {{Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.02005},
}
Figure