Integrated image-based deep learning and language models for primary diabetes care

Jiajia Li, Zhouyu Guan, Jing Wang, Carol Y. Cheung, Yingfeng Zheng, Lee‐Ling Lim, Cynthia Ciwei Lim, Paisan Ruamviboonsuk, Rajiv Raman, Leonor Corsino, Justin B. Echouffo‐Tcheugui, Andrea O. Y. Luk, Li Jia Chen, Xiaodong Sun, Haslina Hamzah, Qiang Wu, Xiangning Wang, Ruhan Liu, Ya Xing Wang, Ting‐Li Chen, Xiao Zhang, Xiaolong Yang, Jun Yin, Jing Wan, Du Wei, Ten Cheer Quek, Jocelyn Hui Lin Goh, Dawei Yang, Xiaoyan Hu, Truong Nguyen, Simon Szeto, Peranut Chotcomwongse, Rachid Malek, Nargiza Normatova, Nilufar Ibragimova, Ramyaa Srinivasan, Pingting Zhong, Wenyong Huang, Chenxin Deng, Lei Ruan, Cuntai Zhang, Chenxi Zhang, Yan Zhou, Chan Wu, Rongping Dai, Sky Wei Chee Koh, Adina Abdullah, Nicholas Ken Yoong Hee, Hong Chang Tan, Zhong Hong Liew, Carolyn Shan‐Yeu Tien, Shih Ling Kao, Amanda Yuan Ling Lim, Shao Feng Mok, Lina Sun, Jing Gu, Liang Wu, Tingyao Li, Di Cheng, Zheyuan Wang, Yiming Qin, Ling Dai, Ziyao Meng, Jia Shu, Yuwei Lu, Nan Jiang, Tingting Hu, Shan Huang, Gengyou Huang, Shujie Yu, Dan Liu, Weizhi Ma, Minyi Guo, Xinping Guan, Xiaokang Yang, Covadonga Bascarán, Charles R Cleland, Yuqian Bao, Elif I. Ekinci, Alicia J. Jenkins, Juliana C.n. Chan, Yong Mong Bee, Sobha Sivaprasad, Jonathan E. Shaw, Rafael Simó, Pearse A. Keane, Ching‐Yu Cheng, Gavin Siew Wei Tan, Weiping Jia, Yih Chung Tham, Huating Li, Bin Sheng, Tien Yin WongNature Medicine 2024 · DOI

Abstract

Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.

Introduction

Diabetes affects over 500 million people globally, with a disproportionate burden in low- and middle-income countries (LMICs). Effective management requires regular screening for diabetic retinopathy (DR)—a leading cause of blindness—and complex, individualized treatment plans. However, a critical shortage of trained primary care physicians (PCPs) and specialists in these regions hinders care delivery. Existing digital solutions are often siloed, focusing either on image-based screening or general health data. DeepDR-LLM aims to bridge this gap by providing an integrated tool that assists with both diagnosis and personalized management.

System Design and Methodology

The DeepDR-LLM system consists of two core modules designed to work in synergy:

  1. DeepDR-Transformer: A Vision Transformer (ViT) model trained for image quality assessment, lesion segmentation (e.g., microaneurysms, hemorrhages), and DR/DME (diabetic macular edema) grading. It is robust enough to handle both high-quality "table-top" fundus images and low-cost portable camera images.
  2. LLM Module: Based on LLaMA-7B, this module was enhanced via Supervised Fine-Tuning (SFT) using a dataset of over 370,000 real-world clinical recommendations. Low-Rank Adaptation (LoRA) and Adapter layers were utilized to maintain efficiency and prevent catastrophic forgetting of general knowledge.
The integrated DeepDR-LLM system architecture, showing the interaction between the LLM and DeepDR-Transformer modules.

The system operates in two modes: a physician-involved mode, where the AI assists the doctor’s judgment, and an automated mode, where the vision module feeds structured data directly into the LLM to generate a management plan.

Evaluation Framework

The authors utilized a massive, multiethnic data infrastructure for training and validation. The LLM module was trained on 371,763 real-world management recommendations. The vision module was trained on 14 datasets for standard images and 7 for portable images, totaling over 1.2 million images across five countries.

Overview of the retrospective and prospective evaluation stages for the DeepDR-LLM system.

The system was benchmarked against "model answers" established by an international panel of 10 experts and further validated in a single-center real-world prospective study involving 769 patients.

Key Results and Clinical Impact

The study demonstrated significant improvements in both diagnostic accuracy and patient outcomes:

  • Diagnostic Accuracy: For identifying referable DR, unassisted PCPs averaged 81.0% accuracy. When assisted by the DeepDR-Transformer, accuracy increased to 92.3%.
  • Clinical Performance: In head-to-head tests, the LLM module’s recommendations were comparable to those of endocrinology residents and significantly outperformed unassisted PCPs in Chinese-language assessments.
  • Real-World Adherence: In the prospective trial, patients in the AI-assisted arm were significantly more likely to follow through on specialist referrals (77.8% vs. 58.4%) and showed better self-management behaviors (diet, exercise, and medication adherence).
  • Empathy and Quality: AI-augmented recommendations were rated as more empathetic and higher quality than those from PCPs alone, suggesting a potential for AI to standardize high-quality "bedside manner."

Discussion

DeepDR-LLM represents a significant step toward democratizing specialist-level diabetes care. By integrating visual findings with clinical metadata, the system provides a holistic management plan that improves patient behavior and clinical adherence.

However, limitations remain. The LLM was primarily fine-tuned on Chinese clinical data, necessitating further validation in diverse global populations. Additionally, while the prospective study showed promising short-term results, long-term follow-up is required to determine the impact on chronic complications like vision loss or kidney failure. Future work must also address hurdles regarding data privacy, legal frameworks, and integration with existing electronic health record (EHR) systems.

Summary updated

Abstract

Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.
BibTeX
@inproceedings{li2024integrated,
  author = {Jiajia Li and Zhouyu Guan and Jing Wang and Carol Y. Cheung and Yingfeng Zheng and Lee‐Ling Lim and Cynthia Ciwei Lim and Paisan Ruamviboonsuk and Rajiv Raman and Leonor Corsino and Justin B. Echouffo‐Tcheugui and Andrea O. Y. Luk and Li Jia Chen and Xiaodong Sun and Haslina Hamzah and Qiang Wu and Xiangning Wang and Ruhan Liu and Ya Xing Wang and Ting‐Li Chen and Xiao Zhang and Xiaolong Yang and Jun Yin and Jing Wan and Du Wei and Ten Cheer Quek and Jocelyn Hui Lin Goh and Dawei Yang and Xiaoyan Hu and Truong Nguyen and Simon Szeto and Peranut Chotcomwongse and Rachid Malek and Nargiza Normatova and Nilufar Ibragimova and Ramyaa Srinivasan and Pingting Zhong and Wenyong Huang and Chenxin Deng and Lei Ruan and Cuntai Zhang and Chenxi Zhang and Yan Zhou and Chan Wu and Rongping Dai and Sky Wei Chee Koh and Adina Abdullah and Nicholas Ken Yoong Hee and Hong Chang Tan and Zhong Hong Liew and Carolyn Shan‐Yeu Tien and Shih Ling Kao and Amanda Yuan Ling Lim and Shao Feng Mok and Lina Sun and Jing Gu and Liang Wu and Tingyao Li and Di Cheng and Zheyuan Wang and Yiming Qin and Ling Dai and Ziyao Meng and Jia Shu and Yuwei Lu and Nan Jiang and Tingting Hu and Shan Huang and Gengyou Huang and Shujie Yu and Dan Liu and Weizhi Ma and Minyi Guo and Xinping Guan and Xiaokang Yang and Covadonga Bascarán and Charles R Cleland and Yuqian Bao and Elif I. Ekinci and Alicia J. Jenkins and Juliana C.N. Chan and Yong Mong Bee and Sobha Sivaprasad and Jonathan E. Shaw and Rafael Simó and Pearse A. Keane and Ching‐Yu Cheng and Gavin Siew Wei Tan and Weiping Jia and Yih Chung Tham and Huating Li and Bin Sheng and Tien Yin Wong},
  title = {{Integrated image-based deep learning and language models for primary diabetes care}},
  booktitle = {Nature Medicine},
  year = {2024},
  doi = {10.1038/s41591-024-03139-8},
}
Figure