TL;DR
This paper introduces DL3M, a novel framework that combines deep learning for medical image classification with large language models (LLMs) for generating expert-level clinical reasoning. By bridging the gap between visual diagnosis and textual explanation, DL3M aims to provide more transparent and reliable medical decision support, though current LLMs still require human oversight.
What Problem Does This Paper Address?
Medical image classifiers are adept at detecting diseases but often fail to explain their reasoning, leaving a critical gap for clinicians. While LLMs can generate text, they struggle with visual reasoning and can produce unstable or incorrect explanations. This disconnect hinders their use in high-stakes medical decisions. The paper addresses this by proposing a unified framework that integrates DL-based image classification with LLM-driven clinical reasoning, aiming to provide both accurate diagnoses and understandable explanations.
Methods
The core of the DL3M framework is a two-stage process. First, a novel, lightweight deep learning architecture called MobileCoAtNet is employed for classifying endoscopic images of gastrointestinal diseases. MobileCoAtNet features a hybrid architecture with two branches that fuse multi-level convolutional features and transformer-based contextual modeling.
Second, the outputs from MobileCoAtNet (the predicted disease class) are fed into various LLMs. These LLMs are then prompted to generate clinical reasoning, including causes, symptoms, treatment, lifestyle advice, and follow-up care. The paper also introduces a structured model selection strategy, evaluating numerous LLMs using a zero-shot prompting approach and then a multi-step reasoning benchmark.
Datasets
The primary dataset used for training and evaluating the DL models is Kvasir-V2, a well-curated collection of 8,000 high-resolution gastrointestinal endoscopy images balanced across eight distinct classes.
The authors also introduce two novel benchmarks for evaluating LLMs:
- GastroLLM-Bench: Used for zero-shot LLM selection, gathering information on causes, symptoms, treatment, lifestyle, and follow-up care for various classes.
- GastroReasonBench: A multi-step reasoning benchmark with multiple-choice questions and justifications, designed to assess LLMs' ability to provide clinically grounded explanations.
Key Insights and Ideas
- The critical role of accurate classification: The study strongly emphasizes that the quality of LLM reasoning is directly dependent on the accuracy of the initial DL-based classification. An incorrect prediction from the DL model inevitably leads to flawed LLM explanations, as LLMs cannot interpret images directly.
- LLMs are not yet autonomous medical reasoners: Despite impressive capabilities, even the best-performing LLMs still exhibit limitations in consistency, prompt sensitivity, and contextual understanding. They require human supervision for reliable use in sensitive medical domains.
- Domain-specific LLMs are crucial: The paper suggests that LLMs trained specifically on gastrointestinal data would likely outperform general-purpose models, leading to more accurate and nuanced reasoning.
- Prompt engineering matters: The performance of LLMs is significantly influenced by the prompt's structure and detail level, highlighting the need for careful prompt design in clinical applications.
Key Results
- MobileCoAtNet Performance: The proposed MobileCoAtNet achieved a high average test accuracy of 97.53% and an F1 score of 97.56% across eight gastrointestinal disease classes, outperforming other tested DL models.
- LLM Performance: Among the 32 LLMs evaluated, GPT-4.1 and GPT-4.1 Mini showed the most adaptive reasoning capabilities, achieving high scores in semantic alignment and contextual consistency. However, no LLM consistently achieved "Human-Like" reasoning status.
- Prompt Sensitivity: LLMs demonstrated varying degrees of stability across different prompt levels. Larger models generally showed more consistent reasoning, while smaller models were more sensitive to prompt wording.
- Commercial vs. Open-Source LLMs: Proprietary LLMs generally outperformed open-source models, particularly in terms of consistency and reasoning quality, suggesting that model scale and training data play a significant role.
Limitations and Future Work
- Limitations:
- Current LLMs still require human supervision for reliable medical reasoning.
- The performance of the entire pipeline is heavily reliant on the accuracy of the initial DL classification.
- LLMs can be sensitive to prompt variations, leading to inconsistent reasoning.
- Smaller LLMs may miss contextual details and struggle with medical nuance.
- Future Work:
- Improve DL-based disease prediction accuracy and incorporate uncertainty estimation.
- Develop domain-adapted LLMs trained specifically on gastrointestinal data.
- Explore structured fine-tuning and reinforcement learning with clinician-verified feedback.
- Develop more robust evaluation setups that go beyond textual similarity to measure factual correctness and clinical validity.
- Work towards real-time clinical decision support systems where LLMs can interact with clinicians and justify their recommendations transparently.
Summary updated