Deep learning system improved detection efficacy of fetal intracranial malformations in a randomized controlled trial

Meifang Lin, Qian Zhou, Ting Lei, Ning Shang, Qiao zheng, Xiaoqin He, Nan Wang, Hongning XieNpj Digit. Medicine 2023 · DOI

TL;DR

This study demonstrates that a deep learning system (PAICS) significantly improves the accuracy of sonologists in detecting fetal intracranial malformations. The system is most efficient when used concurrently during the assessment, offering a promising advancement for prenatal diagnosis.

What Problem Does This Paper Address?

Congenital malformations of the central nervous system are a significant concern in prenatal care. While deep learning (DL) has shown promise in medical image analysis, its real-world impact on the diagnostic performance of clinicians, specifically sonologists, in detecting fetal intracranial malformations has not been thoroughly evaluated. Existing studies often focus on retrospective model performance rather than prospective clinical integration. This paper aims to bridge this gap by assessing the efficacy of a DL system (PAICS) in a randomized controlled trial, comparing its performance in different assistive modes and its impact on sonologists' diagnostic accuracy and efficiency.

Methods

The study employed a three-way crossover, randomized controlled trial design. Thirty-six sonologists of varying expertise levels participated. They reviewed fetal neurosonographic images and videos in three distinct modes:

  1. Unassisted mode: Sonologists reviewed images without any AI assistance.
  2. Concurrent mode: PAICS provided real-time assistance at the beginning of the assessment.
  3. Second mode: PAICS was used after a complete, unaided interpretation by the sonologist.

The PAICS system itself is based on a real-time convolutional neural network (CNN) algorithm (You Only Look Once, version 3 - YOLOv3). The system was trained and validated on a large dataset of fetal neurosonographic images from multiple tertiary hospitals. The trial design ensured that each sonologist experienced all three reading modes, and the order of datasets and modes was randomized to minimize bias. Performance was evaluated using metrics like accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the receiver operating characteristic curve (AUC). Time consumption for each reading mode was also meticulously recorded.

Datasets

The PAICS system was developed and internally validated using neurosonographic images (n=43078) from normal fetuses (n=13400) and fetuses with CNS malformations (n=2448) collected from two tertiary hospitals in China. An external validation dataset (n=812) from a third tertiary hospital was also used.

For the randomized controlled trial, a total of 709 original images/videos (549 images, 160 videos) from 558 fetuses met the inclusion criteria. These were randomly grouped into three datasets for the reading study. The study focused on ten patterns: nine specific fetal intracranial malformations and one normal pattern.

Key Insights and Ideas

  • DL as a powerful diagnostic aid: The study strongly supports the utility of DL systems like PAICS in enhancing diagnostic accuracy for complex medical imaging tasks, moving beyond theoretical performance to demonstrated clinical impact.
  • Concurrent mode efficiency: The concurrent mode, where the DL system assists from the outset, proved to be the most efficient and effective approach, balancing accuracy gains with minimal time overhead.
  • Expertise matters, but DL helps all: While sonologists of all expertise levels (trainees, competent, and experts) benefited from PAICS, the improvements were consistent across the board, suggesting DL can democratize diagnostic capabilities.
  • Localization and diagnosis are key benefits: Sonologists reported that PAICS's ability to localize potential lesions and provide a preliminary diagnosis were the primary drivers of its helpfulness.

Key Results

  • Improved Accuracy: The average accuracy (ACC) of sonologists increased from 73% in the unassisted mode to 80% in the concurrent mode (P < 0.001) and 82% in the second mode (P < 0.001).
  • Enhanced AUC: The area under the curve (AUC) improved from 0.85 (unassisted) to 0.89 (concurrent) and 0.90 (second mode), all statistically significant (P < 0.001).
  • Time Efficiency: While the concurrent mode showed a slight increase in reading time (median 7s vs. 6s), the second mode significantly prolonged reading time (median 11s). This highlights the concurrent mode's superior efficiency.
  • Pattern-Specific Improvements: For most of the ten image patterns, AUCs were significantly improved with PAICS assistance in both concurrent and second modes.
  • Subjective Evaluation: 91.7% of sonologists credited PAICS for lesion localization, and 77.8% for diagnosis. 61% preferred the concurrent mode over the second mode.

Limitations and Future Work

  • Clinical Environment Realism: The study acknowledges that the proportion of abnormal cases was higher than typically seen in real clinical settings, which might affect the generalizability of findings.
  • Sample Size per Pattern: While the overall sample size was substantial, the number of images/videos per specific pattern was relatively limited (around 70), potentially impacting the representation of lesion variations.
  • PAICS Specificity: The study notes that PAICS represents one DL system, and its effectiveness might differ from other emerging DL platforms. Additionally, relatively rare malformations not included in PAICS's training set were not evaluated.
  • Future Work: The authors suggest further research in real-time clinical settings with larger sample sizes to investigate PAICS's assisted efficacy, particularly for recognizing normal cases and a broader range of malformations. They also aim to explore the system's performance in detecting rare conditions.

Summary updated

BibTeX
@article{lin2023deep,
  author = {Meifang Lin and Qian Zhou and Ting Lei and Ning Shang and Qiao zheng and Xiaoqin He and Nan Wang and Hongning Xie},
  title = {{Deep learning system improved detection efficacy of fetal intracranial malformations in a randomized controlled trial}},
  journal = {Npj Digit. Medicine},
  year = {2023},
  doi = {10.1038/s41746-023-00932-6},
}
Figure