The performance of a deep learning system in assisting junior ophthalmologists in diagnosing 13 major fundus diseases: a prospective multi-center clinical trial

Bing Li, Huan Chen, Weihong Yu, Ming Zhang, Fang Lu, Jingxue Ma, Yuhua Hao, Xiaorong Li, Bojie Hu, Lijun Shen, Jianbo Mao, Xixi He, Hao Wang, Dayong Ding, Xirong Li, Youxin ChenNpj Digit. Medicine 2024 · DOI

TL;DR

This study introduces a deep learning system (DLS) designed to assist junior ophthalmologists in diagnosing 13 major fundus diseases. In a prospective multi-center trial, the DLS significantly improved diagnostic consistency by 12% and helped reduce missed diagnoses, demonstrating its potential as a valuable tool in clinical practice.

What Problem Does This Paper Address?

Blindness is a significant global health issue, with age-related macular degeneration, diabetes-related retinopathy, and glaucoma being major contributors. Early detection and timely treatment are crucial for preventing vision loss. However, conventional screening methods are time-consuming, costly, and hampered by a shortage of experienced ophthalmologists, particularly in underdeveloped regions. This creates a need for efficient and accessible diagnostic tools that can augment the capabilities of less experienced clinicians.

Methods

The core of this research is a deep learning system (DLS) built upon a convolutional neural network (CNN) architecture, specifically seResNext50. This DLS has two main components: an image quality assessment model and a diagnostic model. The image quality model acts as a gatekeeper, ensuring that only images of sufficient quality are processed by the diagnostic model. The diagnostic model is designed to simultaneously classify abnormalities and identify specific fundus diseases, employing a multi-label approach. For evaluation, the study introduced a novel metric called "diagnostic consistency," which measures the percentage of images where at least some of the predicted labels match the standard diagnosis, accounting for the possibility of multiple co-existing conditions.

Datasets

The study utilized a large dataset of 81,395 color fundus images for training, validation, and testing the DLS. This dataset was collected from multiple tertiary hospitals. The prospective clinical trial involved 748 patients from five participating hospitals, with a total of 1493 fundus images analyzed.

Key Insights and Ideas

  • DLS as a powerful diagnostic assistant: The DLS doesn't just identify diseases; it actively assists junior ophthalmologists, leading to a significant improvement in their diagnostic accuracy.
  • Reducing missed diagnoses is key: The DLS's primary benefit lies in its ability to help clinicians avoid overlooking diseases, a critical factor in preventing blindness.
  • Diagnostic consistency as a more realistic metric: For conditions that can co-occur, "diagnostic consistency" offers a more practical evaluation of a system's performance than simply looking at individual disease detection.
  • Junior ophthalmologists' trust in AI: The study observed that junior ophthalmologists tended to accept the DLS's suggestions, even when they initially had a different diagnosis, highlighting a growing acceptance of AI in clinical workflows.
  • "Other abnormalities" category is useful: Including a category for conditions not specifically targeted by the DLS allows for a more comprehensive screening and avoids overwhelming clinicians with less common or complex cases that require further specialized evaluation.

Key Results

  • Improved Diagnostic Consistency: The DLS-assisted junior ophthalmologist group (test group) achieved a diagnostic consistency of 84.9%, a significant improvement of approximately 12% compared to the control group (72.9%) (P < 0.001). The DLS group alone also showed high consistency at 85.5%.
  • Enhanced Sensitivity for Specific Diseases: While overall sensitivity for any fundus abnormality was high across groups, the test group showed significantly higher sensitivities for individual diseases (72.2%–100.0%) compared to the control group (50.0%–100.0%).
  • Comparable Specificity: The DLS-assisted group maintained comparable specificities (90.8%–98.7%) to the control group (96.7%–99.8%), indicating a good balance between detecting disease and avoiding false positives.
  • Reduced Missed Diagnoses: The DLS helped junior ophthalmologists correct incorrect labels more often than they changed correct labels to incorrect ones, suggesting a reduction in missed diagnoses.

Limitations and Future Work

  • Dataset Bias: The current dataset primarily consists of Chinese Han patients, limiting generalizability to other ethnicities. Future work should aim to enlarge prospective datasets with more diverse populations.
  • Peripheral Lesions: The DLS may not detect diseases originating in the peripheral retina (e.g., RD, RP) if the fundus images do not capture these areas. The authors suggest that adopting widefield color fundus photography could address this.
  • Educational Potential: The significant improvement in junior ophthalmologists' diagnostic capabilities suggests a potential for the DLS to be used for educational purposes, which warrants further investigation.
  • "Other Abnormalities" Classification: While useful, the "other abnormalities" category is broad. Further refinement and potential sub-classification of these conditions could enhance the DLS's utility.

Summary updated

BibTeX
@article{li2024performance,
  author = {Bing Li and Huan Chen and Weihong Yu and Ming Zhang and Fang Lu and Jingxue Ma and Yuhua Hao and Xiaorong Li and Bojie Hu and Lijun Shen and Jianbo Mao and Xixi He and Hao Wang and Dayong Ding and Xirong Li and Youxin Chen},
  title = {{The performance of a deep learning system in assisting junior ophthalmologists in diagnosing 13 major fundus diseases: a prospective multi-center clinical trial}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-023-00991-9},
}
Figure