Longitudinal Screening for Diabetic Retinopathy in a Nationwide Screening Program: Comparing Deep Learning and Human Graders

Jirawut Limwattanayingyong, Variya Nganthavee, Kasem Seresirikachorn, Tassapol Singalavanija, Ngamphol Soonthornworasiri, Varis Ruamviboonsuk, Chetan Rao, Rajiv Raman, Andrzej Grzybowski, Mike Schaekermann, Lily H. Peng, Dale R. Webster, Christopher Semturs, Jonathan Krause, Rory Sayres, Fred Hersch, Richa Tiwari, Yun Liu, Paisan RuamviboonsukJournal of Diabetes Research 2020 · DOI

TL;DR

This study compares deep learning (DL) and human graders (HG) for longitudinal diabetic retinopathy (DR) screening over two years. While both methods showed decreased sensitivity in the second screening due to a shift in the patient cohort (those with severe DR being referred for treatment), DL consistently maintained a significantly lower false negative rate, making it a promising tool for efficient and effective DR screening programs.

What Problem Does This Paper Address?

Diabetic retinopathy (DR) is a leading cause of preventable blindness, and early detection through regular screening is crucial. Nationwide screening programs aim to identify individuals with sight-threatening DR (STDR) for timely treatment. However, as patients with STDR are referred out of the screening program, subsequent screenings in the same cohort will naturally see a decrease in STDR prevalence. This shift in the patient population can make follow-up screenings more challenging, potentially leading to a decrease in the sensitivity of screening tools. The paper investigates how deep learning (DL) algorithms, increasingly used for DR screening, perform in this longitudinal setting compared to traditional human graders (HG), specifically addressing whether DL can maintain its efficacy and efficiency over time as the screening population evolves.

Methods

The researchers utilized data from a nationwide DR screening program in Thailand, analyzing color fundus photographs from patients with diabetes who underwent screening twice, two years apart. The reference standard for DR grading was established by a panel of three international retina specialists.

The study simulated a real-world screening scenario:

  • First Screening: All patients were screened by both DL and HG.
  • Simulated Referral: Patients identified as having STDR by either DL or HG were considered "referred out" for treatment and excluded from the second screening in their respective arms. This mimics the clinical workflow where individuals needing immediate attention are removed from the screening pool.
  • Second Screening: The remaining patients (those not identified as STDR in the first screening) were re-screened.

The study then compared the performance metrics (sensitivity, specificity, positive predictive value, negative predictive value, accuracy) of DL and HG in both the first and second screenings. They also analyzed the prevalence of STDR and the rates of true positives, false positives, true negatives, and false negatives.

Datasets

The study used a dataset of color fundus photographs from a nationwide DR screening program in Thailand. This dataset comprised 5,738 patients who underwent DR screening twice, approximately two years apart (2014-2016 or 2015-2017). The data was deidentified. No new benchmark datasets were introduced; rather, existing data from a real-world screening program was utilized for analysis.

Key Insights and Ideas

  • Longitudinal Screening Challenges: Follow-up screenings in longitudinal DR screening programs are inherently more difficult. As individuals with obvious STDR are referred for treatment, the remaining cohort may present with more subtle or early-stage DR, making detection harder for both DL and human graders.
  • DL's Advantage in False Negatives: A critical finding is that DL consistently produced significantly fewer false negatives compared to human graders. This is crucial because false negatives represent missed opportunities for treatment and potential vision loss.
  • STDR Prevalence Drop: As expected, the prevalence of STDR significantly decreased in the second screening for both DL and HG arms due to the removal of true positive cases.
  • Sensitivity Decline: Both DL and HG experienced a decrease in sensitivity in the second screening, likely due to the shift in the case spectrum towards more subtle DR. However, DL's sensitivity decline was less pronounced.
  • DME as a Major Component: Diabetic macular edema (DME) was the predominant cause of STDR in both screenings, highlighting its significance in DR management.

Key Results

  • First Screening: DL achieved a sensitivity of 95.03% and HG achieved 73.72%. Specificity was high for both (DL: 97.97%, HG: 98.59%).
  • Second Screening: Sensitivity decreased for DL to 90.05% (p=0.008) and for HG to 57.09% (p<0.001). Specificity remained high for both.
  • False Negative Rates: The rate of false negatives for DL was consistently much lower than for HG, approximately a fifth (0.5-0.6% for DL vs. 2.9-3.2% for HG) across both screenings.
  • STDR Prevalence: STDR prevalence dropped from 12.3% in the first screening to 5.1% for the DL arm and 6.8% for the HG arm in the second screening.
  • Patient Numbers: After simulated referral and excluding ungradable cases, 4,148 patients remained for the second screening in the DL arm, and 4,263 in the HG arm.

Limitations and Future Work

  • Retrospective Nature: The study was retrospective, meaning the cohort was selected based on having photographs from two screenings. This might not perfectly represent real-world screening populations where new patients are continuously entering the program.
  • HG Limitations: Human graders in this study did not have access to previous images, which is a common practice in real-world settings and could have influenced their performance.
  • AI for Cataracts: The current DL model's performance on eyes with cataracts was not assessed. Future work could focus on developing AI that can accurately detect DR in the presence of cataracts, reducing overall referral burdens.
  • Cost-Effectiveness Analysis: The authors suggest that their detailed data could be valuable for future health-economics analyses to guide decisions on whether expert overreading of all "screen-positive" cases is necessary.
  • Broader AI Capabilities: The DL model was designed for DR and DME detection. Future research could explore AI systems capable of detecting multiple retinal conditions simultaneously.
  • Longer-Term Follow-up: The study focused on a 2-year interval. Analyzing longer-term longitudinal trends would provide further insights.

Summary updated

Abstract

On 2-year longitudinal follow-up of a DR screening cohort, STDR prevalence decreased for both DL- and HG-based screening. Follow-up screenings in longitudinal DR screening can be more difficult and induce lower sensitivity for both DL and HG, though the false negative rate was substantially lower for DL. Our data may be useful for health-economics analyses of longitudinal screening settings.
BibTeX
@article{limwattanayingyong2020longitudinal,
  author = {Jirawut Limwattanayingyong and Variya Nganthavee and Kasem Seresirikachorn and Tassapol Singalavanija and Ngamphol Soonthornworasiri and Varis Ruamviboonsuk and Chetan Rao and Rajiv Raman and Andrzej Grzybowski and Mike Schaekermann and Lily H. Peng and Dale R. Webster and Christopher Semturs and Jonathan Krause and Rory Sayres and Fred Hersch and Richa Tiwari and Yun Liu and Paisan Ruamviboonsuk},
  title = {{Longitudinal Screening for Diabetic Retinopathy in a Nationwide Screening Program: Comparing Deep Learning and Human Graders}},
  journal = {Journal of Diabetes Research},
  year = {2020},
  doi = {10.1155/2020/8839376},
}
Figure