Deep Multimodal Guidance for Medical Image Classification

Mayur Mallya, Ghassan Hamarneh

In clinical diagnostics, there is a persistent trade-off between accuracy and accessibility. While high-fidelity modalities like histopathology or advanced MRI sequences provide definitive diagnostic information, they are often invasive, expensive, or time-consuming. Conversely, more accessible modalities like standard clinical photos or basic MRI scans are easier to obtain but may lack the granular detail necessary for high-confidence diagnosis.

Researchers at Simon Fraser University have introduced DeepGuide, a framework designed to resolve this tension. By leveraging "superior" (expensive/invasive) data during training to "teach" a model how to extract richer information from "inferior" (cheap/accessible) data, the system achieves near-expert performance at test time using only the accessible scans.

The DeepGuide Framework

The core of the proposed approach is a three-stage "Deep Multimodal Guidance" pipeline that moves beyond simple label-based knowledge distillation to focus on latent feature representations.

  1. Independent Expert Training: Separate encoder-decoder models are trained for the inferior modality ($I$) and the superior modality ($S$). These models learn to map their respective inputs to a shared diagnostic space.
  2. Guidance Model ($G$): With the expert encoders frozen, a lightweight "Guidance" network is trained to map the latent features of the inferior modality to those of the superior modality. Essentially, $G$ learns to predict what a high-resolution biopsy feature would look like given only a standard MRI feature.
  3. Guided Inference: During inference, the model processes the inferior image to generate its own features while simultaneously using the Guidance model to predict the corresponding superior features. These two feature sets are concatenated to produce a final, more informed diagnostic prediction.
Schematic of the Deep Multimodal Guidance (DeepGuide) framework.

Experimental Evaluation

The authors validated DeepGuide on two distinct clinical challenges: brain tumor classification (RadPath 2020) and skin lesion detection (Derm7pt).

Brain Tumor Classification (RadPath)

The RadPath dataset consists of 221 pairs of 3D MRI sequences (inferior) and digitized histopathology (superior) for glioma classification. The results demonstrate that the guided model significantly outperforms baseline models that rely solely on MRI data. Notably, for the FLAIR MRI sequence, the guidance mechanism provided a +21.3% boost in Balanced Accuracy (BA).

Performance comparison on the RadPath dataset across different MRI sequences.

Skin Lesion Detection (Derm7pt)

In the Derm7pt dataset, which pairs clinical smartphone-style photos with dermoscopic images, the guided model improved performance across all "7-point criteria" for melanoma detection. The overall AUROC increased by +1.6%, demonstrating the framework's versatility across different medical domains.

Key Insights and Discussion

The success of DeepGuide rests on several critical observations:

  • Latent Feature Distillation: Distilling the internal representation of a superior model is more effective than standard distillation (which only mimics final outputs). It allows the student model to inherit the descriptive "vocabulary" of the more advanced modality.
  • Complementary Information: The model performs best when combining original inferior features with predicted superior features. This suggests that even "inferior" modalities contain unique diagnostic signals that are not present in the superior data.
  • Clinical Impact: For brain tumors, the guided MRI-only model actually outperformed a model trained on histopathology alone. This indicates that AI can identify non-invasive patterns in standard scans that are as diagnostic as microscopic tissue analysis, potentially reducing the need for invasive biopsies.

Conclusion and Future Work

DeepGuide offers a computationally efficient way to upgrade the diagnostic power of accessible medical imaging. By acting as a "latent translator," the framework allows clinics to provide high-end diagnostic insights without the associated costs or patient risks of invasive procedures. Future research will focus on improving the reconstruction accuracy of the guidance mapping and extending the framework to handle a continuous stream of new modalities in a clinical pipeline.

Summary updated

Abstract

Medical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice of imaging modality for a particular theranostic task typically involves trade-offs between the feasibility of using a particular modality (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) and the expected performance on a clinical task (e.g., diagnostic accuracy, efficacy of treatment planning and guidance). In this work, we aim to apply the knowledge learned from the less feasible but better-performing (superior) modality to guide the utilization of the more-feasible yet under-performing (inferior) modality and steer it towards improved performance. We focus on the application of deep learning for image-based diagnosis. We develop a light-weight guidance model that leverages the latent representation learned from the superior modality, when training a model that consumes only the inferior modality. We examine the advantages of our method in the context of two clinical applications: multi-task skin lesion classification from clinical and dermoscopic images and brain tumor classification from multi-sequence magnetic resonance imaging (MRI) and histopathology images. For both these scenarios we show a boost in diagnostic performance of the inferior modality without requiring the superior modality. Furthermore, in the case of brain tumor classification, our method outperforms the model trained on the superior modality while producing comparable results to the model that uses both modalities during inference.
BibTeX
@inproceedings{mallya2022deep,
  author = {Mayur Mallya and Ghassan Hamarneh},
  title = {{Deep Multimodal Guidance for Medical Image Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16449-1\_29},
}
Figure