Computer-aided diagnosis has seen significant advancements through self-supervised pre-training. However, existing models, particularly those utilizing Masked Autoencoders (MAEs), often struggle with two critical issues: the semantic gap and modality heterogeneity. The Unified Medical Multi-modal Diagnostic (UMD) framework addresses these by introducing a multi-level reconstruction strategy and a specialized tuning process designed for clinical data.
The Challenges in Medical Multi-modal Learning
Traditional MAE-based pre-training typically focuses on "data-level" reconstruction—predicting missing pixels or text tokens. In medical contexts, this often results in models that overfit to low-level noise (like imaging artifacts) rather than capturing high-level diagnostic semantics.
Furthermore, when moving from general pre-training to specific clinical tasks, models face a distribution gap. In multi-modal settings (e.g., image + text), one modality often dominates the learning process, leading to a "lazy" model that ignores complementary information from the weaker modality.
Multi-level Reconstruction Pre-training (MR-Pretrain)
To bridge the semantic gap, UMD employs a Student-Teacher architecture for pre-training. While the student model performs standard data-level reconstruction (predicting masked pixels/tokens), it is also tasked with feature-level reconstruction.
In this setup, a "Teacher" model—updated via Exponential Moving Average (EMA)—processes the original, unmasked data. The student must reconstruct the Teacher's high-level feature representations from masked inputs. This dual-level approach forces the encoder to learn transferable semantic features that are more relevant to actual medical diagnosis than raw pixel statistics.
Heterogeneity-combat Downstream Tuning
To adapt the pre-trained model to specific clinical datasets, UMD introduces two key modules:
- TD-Calib (Task-oriented Distribution Calibration): Before supervised fine-tuning, the model undergoes an unsupervised "warm-up" on the downstream dataset. By performing reconstruction tasks on the target data distribution first, the encoder is better prepared for the final supervised task.
- GM-Coord (Gradient-guided Modality Coordination): This module adaptively modulates gradients during training. If one modality (e.g., text) begins to optimize significantly faster than the other (e.g., images), GM-Coord slows its learning to ensure the model develops a holistic, multi-modal understanding.
Experimental Results
The UMD framework was evaluated across several benchmark datasets, including MedICaT and ROCO for pre-training, and VQA-RAD, SLAKE, and MELINDA for downstream tasks.
- Medical VQA: UMD achieved state-of-the-art results with 78.71% accuracy on VQA-RAD and 84.73% on SLAKE, outperforming previous leaders like M3AE.
- Image-Text Retrieval: On the ROCO dataset, UMD reached a 54.28% R@5 for text-to-image retrieval, demonstrating superior cross-modal alignment.
- Classification: The framework achieved 79.58% accuracy on the MELINDA dataset, confirming its versatility across different diagnostic formats.
Conclusion
By prioritizing semantic feature reconstruction over simple pixel prediction and implementing strategies to balance modality contributions, the UMD framework sets a new standard for medical multi-modal AI. Future research will focus on expanding this framework to 3D imaging (CT/MRI) and longitudinal patient records to further enhance its clinical utility.
Summary updated