Accuracy and feasibility with AI-assisted OCT in retinal disorder community screening

Jianhao Bai, Zhongqi Wan, Ping Li, Lei Chen, Jingcheng Wang, Yu Fan, Xinjian Chen, Qing Peng, Peng GaoFrontiers in Cell and Developmental Biology 2022 · DOI

TL;DR

This study introduces an AI-powered system that uses Optical Coherence Tomography (OCT) to automatically detect 15 different retinal disorders in a community screening setting. The AI demonstrated high accuracy, sensitivity, and specificity, performing comparably to retinal specialists and outperforming junior and senior ophthalmologists, highlighting its potential for efficient and effective early detection of eye diseases.

What Problem Does This Paper Address?

The increasing prevalence of age-related eye diseases, coupled with a shortage of experienced ophthalmologists, creates a significant challenge for timely diagnosis and treatment, especially in community screening programs. While AI has shown promise in analyzing retinal images, most existing systems focus on a single condition. This paper addresses the need for a comprehensive AI solution that can simultaneously detect multiple retinal disorders from OCT scans, making community screening more efficient and accessible.

Methods

The researchers developed an AI system that integrates deep learning algorithms for analyzing OCT images. The pipeline involves:

  1. Quality Assessment: An AI model (ResNet-50 based) filters out OCT scans with poor image quality to ensure reliable analysis.
  2. Object Detection: A deep learning convolutional neural network is used to detect up to 15 categories of retinal pathologies.
  3. Keypoint Detection: A specialized model identifies Bruch's membrane opening locations to refine bounding box predictions, especially near the optic disc.
  4. Auxiliary Diagnosis: A probability score (0-1) is generated based on all detected lesions and their locations, providing an overall diagnostic assessment.

The AI algorithm was built using a multi-stage object detection model (Cascade-RCNN with Feature Pyramid Network) and trained on a dataset of OCT scans. The performance of the AI was then compared against diagnoses made by three groups of ophthalmologists: junior ophthalmologists (OP1), senior ophthalmologists (OP2), and retinal specialists (OP3), who served as the gold standard. Statistical analysis included ROC curves, AUC, sensitivity, specificity, Youden index, and kappa statistics.

Datasets

The study enrolled 477 subjects (954 eyes) from four local communities. After excluding scans with poor image quality, 878 eyes were used for analysis. The paper does not explicitly mention a newly introduced benchmark dataset; rather, it utilized OCT scans collected during the community screening for both training and evaluation of the AI model. The dataset was split into training, validation, and testing sets with a ratio of 6:2:2.

Key Insights and Ideas

  • Multi-Disorder Detection is Key: The most significant insight is the development and validation of an AI system capable of detecting 15 different retinal disorders simultaneously. This moves beyond single-disease AI models and is crucial for real-world community screening where patients often present with multiple conditions.
  • OCT as a Richer Data Source: The study emphasizes the advantage of using OCT scans over fundus photography for AI analysis, as OCT provides detailed cross-sectional information of retinal layers, enabling the detection of a wider range of pathologies.
  • AI as a Specialist's Ally: The AI's performance was closest to that of the retinal specialists, suggesting it can act as a powerful assistive tool, potentially augmenting the capabilities of even experienced clinicians and significantly improving the efficiency of screening.
  • Feasibility in Real-World Settings: The study successfully implemented and evaluated the AI system within a community screening context, demonstrating its practical feasibility and efficiency, with OCT scans taking an average of 18.4 seconds per eye and AI report generation in about 4 minutes.

Key Results

  • High Overall Accuracy: The AI system achieved a large overall Area Under the Receiver Operating Characteristic Curve (AUC) ranging from 0.891 to 0.997 when compared to retinal specialists.
  • Excellent Sensitivity and Specificity: The AI demonstrated high sensitivity (87.65-100%) and high specificity (80.12-99.41%) across the 14 detectable retinal disorders.
  • Superior Performance: The AI's AUC was significantly larger than that of junior (OP1) and senior (OP2) ophthalmologists when compared to the retinal specialists (OP3), indicating its results were closer to the gold standard.
  • Specific Disorder Performance: The AI showed exceptional performance in detecting Cystoid Macular Edema (CME) and Retinal Hemorrhage, with AUCs of 0.997 and 0.991, respectively.
  • Kappa Statistics: The average kappa value for AI vs. retinal specialists (OP3) was 0.731, indicating substantial agreement, which was higher than AI vs. junior ophthalmologists (0.579) and AI vs. senior ophthalmologists (0.707).
  • Efficiency: The average OCT scan time was 18.4 seconds per eye, and the AI report generation time was approximately 4 minutes.

Limitations and Future Work

  • Single OCT Instrument and Location: The study used OCT images from a single instrument and was conducted in one medical center within a specific district. The authors acknowledge that further validation is needed with images from different OCT devices and across diverse communities to certify the AI's generalizability.
  • Limited Retinal Detachment (RD) Cases: No cases of retinal detachment were detected in the study population, preventing a thorough evaluation of the AI's performance for this specific disorder. Further studies with larger cohorts including RD cases are suggested.
  • Algorithm Improvement: For certain retinal disorders, the specificity and sensitivity of the AI still require improvement, suggesting ongoing algorithm refinement is necessary.
  • External Validation: While an external test set was used, further validation with larger, more diverse external datasets would strengthen the AI's generalizability.

Summary updated

Abstract

<b>Objective:</b> To evaluate the accuracy and feasibility of the auto-detection of 15 retinal disorders with artificial intelligence (AI)-assisted optical coherence tomography (OCT) in community screening. <b>Methods:</b> A total of 954 eyes of 477 subjects from four local communities were enrolled in this study from September to December 2021. They received OCT scans covering an area of 12 mm × 9 mm at the posterior pole retina involving the macular and optic disc, as well as other ophthalmic examinations performed using their demographic information recorded. The OCT images were analyzed using integrated software with the previously established algorithm based on the deep-learning method and trained to detect 15 kinds of retinal disorders, namely, pigment epithelial detachment (PED), posterior vitreous detachment (PVD), epiretinal membranes (ERMs), sub-retinal fluid (SRF), choroidal neovascularization (CNV), drusen, retinoschisis, cystoid macular edema (CME), exudation, macular hole (MH), retinal detachment (RD), ellipsoid zone disruption, focal choroidal excavation (FCE), choroid atrophy, and retinal hemorrhage. Meanwhile, the diagnosis was also generated from three groups of individual ophthalmologists (group of retina specialists, senior ophthalmologists, and junior ophthalmologists) and compared with those by the AI. The area under the receiver operating characteristic curve (AUC), sensitivity, and specificity were calculated, and kappa statistics were performed. <b>Results:</b> A total of 878 eyes were finally enrolled, with 76 excluded due to poor image quality. In the detection of 15 retinal disorders, the ROC curve comparison between AI and professors' presented relatively large AUC (0.891-0.997), high sensitivity (87.65-100%), and high specificity (80.12-99.41%). Among the ROC curve comparisons with those by the retina specialists, AI was the closest one to the professors' compared to senior and junior ophthalmologists (<i>p</i> < 0.05). <b>Conclusion:</b> AI-assisted OCT is highly accurate, sensitive, and specific in auto-detection of 15 kinds of retinal disorders, certifying its feasibility and effectiveness in community ophthalmic screening.
BibTeX
@inproceedings{bai2022accuracy,
  author = {Jianhao Bai and Zhongqi Wan and Ping Li and Lei Chen and Jingcheng Wang and Yu Fan and Xinjian Chen and Qing Peng and Peng Gao},
  title = {{Accuracy and feasibility with AI-assisted OCT in retinal disorder community screening}},
  booktitle = {Frontiers in Cell and Developmental Biology},
  year = {2022},
  doi = {10.3389/fcell.2022.1053483},
}
Figure