Abstract
Medical diagnosis increasingly relies on diverse data modalities, yet integrating these into a unified artificial intelligence framework remains challenging due to data scarcity and "modality domination." This paper introduces AdaCoMed (Adaptive Collaborative-learning Medical framework), a novel approach that harmonizes the deep feature extraction capabilities of single-modality large models with the specialized cross-modal reasoning of smaller, task-specific models. By employing an adaptive weighting strategy and contrastive alignment, AdaCoMed achieves state-of-the-art performance in clinical tasks while maintaining exceptional computational efficiency.
Introduction
Effective medical diagnosis necessitates a holistic view of patient data, including radiology images, pathology slides, clinical notes, and electronic health records (EHR). However, developing Large Multi-modal Models (LMMs) for healthcare is hindered by the lack of massive, modality-complete datasets. Current approaches typically fall into two categories: single-modal large models, which excel at specific feature extraction but lack cross-modal context, and multi-modal small models, which understand inter-modality relationships but suffer from limited capacity.
A significant hurdle in combining these models is modality domination, where a single high-information modality (often text) overshadows others during training, leading the model to ignore critical visual or temporal cues. AdaCoMed addresses these challenges by facilitating a collaborative learning environment between large and small models.
Methodology
AdaCoMed utilizes a dual-pathway architecture designed to leverage the strengths of both large and small models.
Dual-Pathway Collaboration
- Large Model Pathway: This pathway utilizes frozen, specialized large models for various modalities (e.g., X-ray, CT, Pathology). Features are integrated via a Mixture-of-Modality-Experts (MoME) fusion module, providing flexibility for missing data scenarios.
- Small Model Pathway: A pre-trained, task-driven small model serves as a "mentor," providing well-structured cross-modal patterns learned from specific medical tasks.
Adaptive Weighting and Contrastive Alignment
To prevent modality domination, AdaCoMed employs a Contrastive Loss to align the large model's features with the small model's structured representations. The framework dynamically calculates a similarity score between the two pathways. Depending on this score, it applies one of several adaptive weighting strategies (Learnable, Inverse Contrastive, Gaussian, or Threshold) to balance the influence of each pathway during training.
Experimental Results
The framework was evaluated on two real-world datasets: MIMIC-IV-MM (X-rays, text, EHR) and MMIST ccRCC (CT, pathology, clinical data).
Performance and Efficiency
AdaCoMed demonstrated substantial improvements over existing benchmarks:
- Accuracy: On the MIMIC-IV-MM mortality task, AUROC improved by 3.1% over ensemble methods and 57.5% over unified models like OneLLM. For kidney cancer survival prediction (MMIST ccRCC), it achieved a 15.4% improvement in AUROC.
- Computational Efficiency: The model requires only 21.3 million trainable parameters. Training time per epoch was reduced to 0.02 minutes, significantly faster than competing models which required up to 74 minutes.
Visualizing Feature Fusion
UMAP visualizations confirm that AdaCoMed effectively mitigates modality domination. While direct combination often leads to fused features overlapping almost entirely with text, AdaCoMed creates a distinct, integrated representation that incorporates visual and temporal signals.
Conclusion
AdaCoMed represents a significant step forward in multi-modal medical AI by treating small, task-specific models as guides for large-scale feature extractors. This collaboration allows for high-performance diagnosis even with limited multi-modal data. Future work will focus on expanding the framework to include more diverse modalities such as genomics and audio, further scaling the collaborative learning paradigm.
Summary updated