Abstract
Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.
Introduction
Diabetes affects over 500 million people globally, with a disproportionate burden in low- and middle-income countries (LMICs). Effective management requires regular screening for diabetic retinopathy (DR)—a leading cause of blindness—and complex, individualized treatment plans. However, a critical shortage of trained primary care physicians (PCPs) and specialists in these regions hinders care delivery. Existing digital solutions are often siloed, focusing either on image-based screening or general health data. DeepDR-LLM aims to bridge this gap by providing an integrated tool that assists with both diagnosis and personalized management.
System Design and Methodology
The DeepDR-LLM system consists of two core modules designed to work in synergy:
- DeepDR-Transformer: A Vision Transformer (ViT) model trained for image quality assessment, lesion segmentation (e.g., microaneurysms, hemorrhages), and DR/DME (diabetic macular edema) grading. It is robust enough to handle both high-quality "table-top" fundus images and low-cost portable camera images.
- LLM Module: Based on LLaMA-7B, this module was enhanced via Supervised Fine-Tuning (SFT) using a dataset of over 370,000 real-world clinical recommendations. Low-Rank Adaptation (LoRA) and Adapter layers were utilized to maintain efficiency and prevent catastrophic forgetting of general knowledge.
The system operates in two modes: a physician-involved mode, where the AI assists the doctor’s judgment, and an automated mode, where the vision module feeds structured data directly into the LLM to generate a management plan.
Evaluation Framework
The authors utilized a massive, multiethnic data infrastructure for training and validation. The LLM module was trained on 371,763 real-world management recommendations. The vision module was trained on 14 datasets for standard images and 7 for portable images, totaling over 1.2 million images across five countries.
The system was benchmarked against "model answers" established by an international panel of 10 experts and further validated in a single-center real-world prospective study involving 769 patients.
Key Results and Clinical Impact
The study demonstrated significant improvements in both diagnostic accuracy and patient outcomes:
- Diagnostic Accuracy: For identifying referable DR, unassisted PCPs averaged 81.0% accuracy. When assisted by the DeepDR-Transformer, accuracy increased to 92.3%.
- Clinical Performance: In head-to-head tests, the LLM module’s recommendations were comparable to those of endocrinology residents and significantly outperformed unassisted PCPs in Chinese-language assessments.
- Real-World Adherence: In the prospective trial, patients in the AI-assisted arm were significantly more likely to follow through on specialist referrals (77.8% vs. 58.4%) and showed better self-management behaviors (diet, exercise, and medication adherence).
- Empathy and Quality: AI-augmented recommendations were rated as more empathetic and higher quality than those from PCPs alone, suggesting a potential for AI to standardize high-quality "bedside manner."
Discussion
DeepDR-LLM represents a significant step toward democratizing specialist-level diabetes care. By integrating visual findings with clinical metadata, the system provides a holistic management plan that improves patient behavior and clinical adherence.
However, limitations remain. The LLM was primarily fine-tuned on Chinese clinical data, necessitating further validation in diverse global populations. Additionally, while the prospective study showed promising short-term results, long-term follow-up is required to determine the impact on chronic complications like vision loss or kidney failure. Future work must also address hurdles regarding data privacy, legal frameworks, and integration with existing electronic health record (EHR) systems.
Summary updated