Unified Multi-Modal Diagnostic Framework With Reconstruction Pre-Training and Heterogeneity-Combat Tuning

Yupei Zhang, Li Pan, Qiushi Yang, Tan Li, Zhen Chen

TL;DR

The Unified Medical Multi-modal Diagnostic (UMD) framework improves computer-aided diagnosis by combining a novel "multi-level" reconstruction pre-training strategy with "heterogeneity-combat" tuning. By forcing models to reconstruct high-level semantic features (not just raw pixels) and balancing the influence of different data modalities during fine-tuning, UMD sets new state-of-the-art benchmarks across medical VQA, image-text retrieval, and classification tasks.

What Problem Does This Paper Address?

While self-supervised pre-training has revolutionized medical AI, existing methods—particularly those based on Masked Autoencoders (MAEs)—face two major hurdles:

  1. The Semantic Gap: Most MAEs focus on "data-level" reconstruction (predicting missing pixels or text tokens). This often leads the model to overfit on low-level details (like background noise or lighting) rather than capturing the high-level medical semantics necessary for diagnosis.
  2. The Heterogeneity Challenge: There is often a massive distribution gap between general pre-training data and specific downstream clinical datasets. Furthermore, in multi-modal tasks (image + text), one modality often "dominates" the learning process, causing the model to ignore useful information from the other.

Methods

The UMD framework introduces two primary stages to solve these issues:

1. Multi-level Reconstruction Pre-training (MR-Pretrain)

Instead of just predicting pixels, UMD uses a Student-Teacher architecture.

  • Data-level: The student model reconstructs masked pixels and tokens.
  • Feature-level: A "Teacher" model (updated via Exponential Moving Average) processes the original, unmasked data. The student is then tasked with reconstructing the Teacher's high-level feature representations from masked inputs. This forces the student to learn "transferable semantic features" rather than just local statistics.

2. Heterogeneity-combat Downstream Tuning

To bridge the gap to specific clinical tasks, the authors propose:

  • TD-Calib (Task-oriented Distribution Calibration): Before supervised fine-tuning, the model performs an unsupervised "warm-up" on the downstream dataset using the same reconstruction tasks. This helps the encoder adapt to the specific data distribution of the new task.
  • GM-Coord (Gradient-guided Modality Coordination): This module monitors the "contribution" of images vs. text. If one modality is optimizing too quickly and suppressing the other, GM-Coord adaptively modulates the gradients, ensuring both modalities are effectively utilized.

Datasets

The authors utilized a robust suite of public medical datasets:

  • Pre-training: MedICaT (217k+ images/captions) and ROCO (81k+ radiology images).
  • Downstream Evaluation:
    • VQA: VQA-RAD, SLAKE, and VQA-Med-2019.
  • Retrieval: ROCO (image-to-text and text-to-image).
  • Classification: MELINDA (biomedical experiment method classification).

Key Insights and Ideas

  • Semantics > Pixels: The "aha moment" here is that raw data reconstruction is a "noisy" signal for medical tasks. By using a teacher model to provide a "semantic target," the pre-training becomes much more relevant to actual diagnostic needs.
  • The Distribution Bridge: Simply fine-tuning on a small labeled dataset isn't enough. The TD-Calib stage shows that "re-acquainting" the model with the target data's distribution in an unsupervised way significantly boosts final performance.
  • Modality Fairness: In medical VQA, models often become "lazy" and rely solely on text or solely on images. GM-Coord acts as a referee, slowing down the "fast" modality to let the "slow" one catch up, leading to a more holistic understanding.

Key Results

UMD consistently outperformed state-of-the-art models like M3AE, CLIP, and FLIP:

  • Medical VQA: Achieved an overall accuracy of 78.71% on VQA-RAD and 84.73% on SLAKE, outperforming the previous best (M3AE) by 1.70% and 1.48% respectively.
  • Image-Text Retrieval: On the ROCO dataset, UMD improved R@5 (Recall at 5) for text-to-image retrieval to 54.28%, a significant jump over baseline methods.
  • Classification: On the MELINDA dataset, UMD reached 79.58% accuracy, proving its versatility across different diagnostic formats.

Limitations and Future Work

The authors note that while UMD is powerful, it currently relies on a specific set of pre-training objectives. Future work could explore:

  • Broader Modalities: Expanding the framework to include 3D volumes (CT/MRI) or longitudinal patient records.
  • Computational Efficiency: The student-teacher setup and the multi-stage tuning process (TD-Calib followed by GM-Coord) add computational overhead compared to standard fine-tuning.
  • Clinical Integration: Testing the framework in real-world clinical workflows to see how it assists human radiologists in practice.

Summary updated

BibTeX
@article{zhang2025unified,
  author = {Yupei Zhang and Li Pan and Qiushi Yang and Tan Li and Zhen Chen},
  title = {{Unified Multi-Modal Diagnostic Framework With Reconstruction Pre-Training and Heterogeneity-Combat Tuning}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2025},
  doi = {10.1109/jbhi.2024.3384407},
}
Figure