Multimodal Medical Image Classification via Synergistic Learning Pre-training

Qinghua Lin, Guanghai Liu, Zuoyong Li, Yang Li, Yuting Jiang, Xiang Wu

TL;DR

The authors propose MICS, a "pre-training + fine-tuning" framework for multimodal medical image classification that thrives even when expert labels are scarce. By treating different imaging modalities (like white light vs. narrow-band imaging) as "natural augmentations" of each other and using a novel Shift Vector Dictionary to perturb features during fine-tuning, MICS significantly outperforms state-of-the-art semi-supervised methods in diagnosing gastrointestinal diseases.

What Problem Does This Paper Address?

In clinical settings, doctors often use multiple imaging modalities—such as standard White Light Imaging (WLI) and Narrow-Band Imaging (NBI)—to get a complete picture of a lesion. However, building AI models for this task faces two major hurdles:

  1. Label Scarcity: Expert annotation is expensive and time-consuming. Most medical datasets have very few labeled samples.
  2. Augmentation Pitfalls: Standard data augmentations used in computer vision (like heavy cropping or color jittering) can destroy the subtle pathological features essential for diagnosis.

Existing semi-supervised learning (SSL) methods struggle to fuse these modalities effectively when data is limited, often leading to overfitting or high prediction uncertainty.

Methods

The researchers introduced MICS (Multimodal Medical Image Classification via Synergistic Learning), which operates in three distinct stages:

1. Synergistic Learning Pre-training (Self-Supervised)

Using unlabeled data, the model learns robust representations through three "synergistic" tasks:

  • Consistency Learning: It treats one modality as a "strongly augmented" version of the other, forcing the model to recognize they represent the same underlying pathology.
  • Reconstructive Learning: Using a masked autoencoder approach, the model learns to reconstruct original images from partial patches, capturing fine-grained local details.
  • Aligned Learning: It uses contrastive learning (InfoNCE loss) to align high-dimensional features from different modalities in a shared latent space.

2. Shift Vector Dictionary (SVD) Construction

After pre-training, the model clusters the learned features using K-means to build a dictionary of "shift vectors." These vectors represent the typical distribution differences between classes and modalities.

3. Multimodal Fusion Fine-tuning

During the final stage, the model uses a small amount of labeled data. To prevent overfitting, it performs implicit augmentation: instead of creating new images, it adds the previously learned shift vectors to the fused features. Finally, it employs Uncertainty-Based Evidential Fusion (TMC) to weigh the reliability of each modality before making a final classification.

Datasets

The study utilized two primary public gastroscopy datasets:

  • Kvasir: 3,000 images across 6 categories (3 normal, 3 abnormal).
  • Kvasirv2: 6,000 images across the same 6 categories.

Note on Modality: Since these datasets originally contained only single-modality images, the authors used a generative model (WtNGAN) to produce paired NBI-like images for every WLI image, creating a synthetic multimodal benchmark.

Key Insights and Ideas

  • Modalities as Natural Augmentations: The most elegant insight is rethinking multimodal data. Instead of trying to find the "perfect" synthetic augmentation (like Mixup), the authors use the inherent differences between WLI and NBI as a clinically relevant augmentation strategy.
  • Feature-Space Perturbation: By using the Shift Vector Dictionary, the model performs "implicit" data augmentation. This fills the gaps in the latent space between modalities without the computational overhead or risk of generating "unrealistic" medical images.
  • Synergy of Local and Global: By combining reconstruction (which focuses on local pixels) with alignment (which focuses on global semantics), the model develops a more holistic "understanding" of medical images than models using only one approach.

Key Results

MICS was tested against several state-of-the-art SSL methods (e.g., FixMatch, FlexMatch, SIAVC) across different label ratios (5%, 10%, 30%, 50%).

  • Superior Accuracy at Low Label Ratios: On the Kvasirv2 dataset with only 5% labeled data, MICS achieved 70.25% accuracy, significantly outperforming the next best method (SIAVC at 61.83%) and traditional FixMatch (49.75%).
  • Robustness: Qualitative UMAP visualizations showed that MICS creates much clearer clusters in the feature space, successfully separating tricky categories like "Normal-Cecum" from "Ulcerative-Colitis," where other models showed significant overlap.
  • Ablation Success: Removing any component (Consistency, Reconstruction, or Alignment) led to a drop in performance, proving that the "synergy" is essential.

Limitations and Future Work

  • Synthetic Modalities: A primary limitation is that the paired NBI images were generated by a GAN rather than captured simultaneously in a clinic. While GANs are sophisticated, they may not perfectly capture the physics of real NBI.
  • Generalization: The current work focuses on gastroscopy. The authors suggest that future work should validate MICS on other multimodal medical tasks, such as combining MRI and CT scans or different staining techniques in pathology.
  • Computational Cost: The three-stage process (pre-train, dictionary build, fine-tune) is more complex than standard end-to-end training, which might be a hurdle for rapid deployment.

Summary updated

Abstract

Multimodal pathological images are usually in clinical diagnosis, but computer vision-based multimodal image-assisted diagnosis faces challenges with modality fusion, especially in the absence of expert-annotated data. To achieve the modality fusion in multimodal images with label scarcity, we propose a novel ``pretraining + fine-tuning" framework for multimodal semi-supervised medical image classification. Specifically, we propose a synergistic learning pretraining framework of consistency, reconstructive, and aligned learning. By treating one modality as an augmented sample of another modality, we implement a self-supervised learning pre-train, enhancing the baseline model's feature representation capability. Then, we design a fine-tuning method for multimodal fusion. During the fine-tuning stage, we set different encoders to extract features from the original modalities and provide a multimodal fusion encoder for fusion modality. In addition, we propose a distribution shift method for multimodal fusion features, which alleviates the prediction uncertainty and overfitting risks caused by the lack of labeled samples. We conduct extensive experiments on the publicly available gastroscopy image datasets Kvasir and Kvasirv2. Quantitative and qualitative results demonstrate that the proposed method outperforms the current state-of-the-art classification methods. The code will be released at: https://github.com/LQH89757/MICS.
BibTeX
@misc{lin2025multimodal,
  author = {Qinghua Lin and Guanghai Liu and Zuoyong Li and Yang Li and Yuting Jiang and Xiang Wu},
  title = {{Multimodal Medical Image Classification via Synergistic Learning Pre-training}},
  year = {2025},
  eprint = {2509.17492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.17492},
}
Figure