A Concept-based Interpretable Model for the Diagnosis of Choroid Neoplasias using Multimodal Data

Yifan Wu, Yang Liu, Yue Yang, Michael S. Yao, Wenli Yang, Xuehui Shi, Lihong Yang, Dongjun Li, Yueming Liu, James C. Gee, Xuan Yang, Wenbin Wei, Shi Gu

Introduction

Diagnosing rare choroid neoplasias (eye cancers) is a high-stakes clinical challenge. With an incidence rate of only 5.1 per million, few clinicians encounter enough cases to become experts. While deep learning offers a potential solution, standard "black-box" models face hurdles in data scarcity and a lack of interpretability. In clinical settings, doctors must verify AI suggestions against their own observations; if a model cannot explain its reasoning in medical terms, its utility remains limited.

To bridge this gap, researchers developed the Multimodal Medical Concept Bottleneck Model (MMCBM). This framework matches the performance of opaque systems while providing actionable explanations that improved junior doctors' diagnostic accuracy by 42%.

Methodology: The MMCBM Framework

The MMCBM utilizes a "Concept Bottleneck" architecture. Instead of mapping images directly to a diagnosis, the model first predicts a set of intermediate, human-understandable clinical concepts, such as "clustered hypofluorescence" or "hemispherical solid lesions."

Overview of the MMCBM framework, illustrating data preprocessing with LLM-based prompting, concept construction via expert verification, and the interpretable prediction pipeline.

Key technical innovations include:

  • Automated Concept Extraction: GPT-4 was used to parse nearly two decades of radiological reports, extracting 103 distinct clinical concepts, which were then refined by senior ophthalmologists.
  • Concept Grounding: High-dimensional image features are mapped to these clinical concepts using Concept Activation Vectors (CAVs) and Support Vector Machines (SVMs).
  • Multimodal Fusion: The model integrates Fluorescein Angiography (FA), Indocyanine Green Angiography (ICGA), and Ocular Ultrasound (US) using an attention-pooling mechanism to weigh the importance of different images.
  • Human-in-the-Loop: The framework allows clinicians to intervene at the bottleneck stage, adjusting the confidence of predicted concepts to refine the final diagnosis.

The Choroid Tri-Modal Imaging (CTI) Dataset

The study introduces the Choroid Tri-Modal Imaging (CTI) dataset, the largest of its kind globally. It comprises data from 750 patients collected between 2004 and 2022.

Composition of the CTI dataset, showing the distribution of pathologies (Melanoma, Hemangioma, Metastatic Carcinoma), imaging modality availability per patient, and the training/test dataset splits.

Unlike general-purpose medical datasets, CTI includes expert-verified image-concept pairs and anonymized radiological reports, providing a rich ground truth for interpretability. The dataset includes cases of choroidal melanoma (542), hemangioma (128), and metastatic carcinoma (80).

Key Results and Insights

  • Interpretability is not a "Tax": MMCBM achieved an F1 score of 0.91, slightly outperforming the black-box baseline (0.89). The clinical concepts act as a form of informed regularization, helping the model navigate class imbalances.
  • AI as an Educational Multiplier: While senior doctors' performance remained stable, junior doctors saw a massive 42% improvement in their F1 scores when shown the model's predicted concepts. This suggests that interpretable AI can serve as a powerful "on-the-job" training tool.
  • Concept Accuracy: SVMs trained to identify FA and ICGA concepts achieved over 90% accuracy on test data, proving that the model's internal logic aligns closely with physical reality.

Conclusion and Future Work

While the results are promising, the authors acknowledge that medical imaging is inherently noisy due to variations in acquisition protocols over an 18-year period. Future work will involve rigorous validation through prospective studies and randomized clinical trials before the model can be deployed in live hospital workflows. Additionally, collaboration with regulatory bodies is necessary to ensure the tool meets safety and equity standards for medical devices.

Summary updated

Abstract

Diagnosing rare diseases presents a common challenge in clinical practice, necessitating the expertise of specialists for accurate identification. The advent of machine learning offers a promising solution, while the development of such technologies is hindered by the scarcity of data on rare conditions and the demand for models that are both interpretable and trustworthy in a clinical context. Interpretable AI, with its capacity for human-readable outputs, can facilitate validation by clinicians and contribute to medical education. In the current work, we focus on choroid neoplasias, the most prevalent form of eye cancer in adults, albeit rare with 5.1 per million. We built the so-far largest dataset consisting of 750 patients, incorporating three distinct imaging modalities collected from 2004 to 2022. Our work introduces a concept-based interpretable model that distinguishes between three types of choroidal tumors, integrating insights from domain experts via radiological reports. Remarkably, this model not only achieves an F1 score of 0.91, rivaling that of black-box models, but also boosts the diagnostic accuracy of junior doctors by 42%. This study highlights the significant potential of interpretable machine learning in improving the diagnosis of rare diseases, laying a groundwork for future breakthroughs in medical AI that could tackle a wider array of complex health scenarios.
BibTeX
@inproceedings{wu2025concept,
  author = {Yifan Wu and Yang Liu and Yue Yang and Michael S. Yao and Wenli Yang and Xuehui Shi and Lihong Yang and Dongjun Li and Yueming Liu and James C. Gee and Xuan Yang and Wenbin Wei and Shi Gu},
  title = {{A Concept-based Interpretable Model for the Diagnosis of Choroid Neoplasias using Multimodal Data}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-58801-7},
}
Figure