Diagnostic Performance of a Deep Learning Model Deployed at a National COVID-19 Screening Facility for Detection of Pneumonia on Frontal Chest Radiographs

Jordan Sim, Yong Han Ting, Yuan Tang, Yangqin Feng, Xiaofeng Lei, Xiaohong Wang, Wenxiang Chen, Su Huang, Sum-Thai Wong, Zhongkang Lu, Yingnan Cui, Soo-Kng Teo, Xinxing Xu, Weimin Huang, Cher-Heng TanHealthcare 2022 · DOI

TL;DR

This paper presents a deep learning model for detecting pneumonia on chest X-rays (CXRs) in COVID-19 suspects, demonstrating its effectiveness in a real-world national screening facility. The deployed model not only maintained high diagnostic performance but also significantly reduced report turnaround time, highlighting its practical utility in high-throughput clinical settings.

What Problem Does This Paper Address?

The COVID-19 pandemic placed immense strain on healthcare systems, necessitating rapid and accurate screening of suspected cases. While RT-PCR is the gold standard for diagnosis, it can be time-consuming. Chest X-rays (CXRs) are a readily available and widely used initial radiological investigation for identifying pneumonia, a key indicator of COVID-19 severity. However, the diagnostic performance of AI models developed in simulated environments often degrades when deployed in real-world clinical workflows. This paper addresses the challenge of developing, validating, and integrating a deep learning model for pneumonia detection on CXRs within a national COVID-19 screening facility, aiming to improve efficiency without compromising accuracy.

Methods

The researchers developed an ensemble deep learning model based on the DenseNet121 architecture. They employed a transfer learning approach, pre-training the model on the large ChestX-ray14 dataset to learn general thoracic disease features. This pre-trained model was then fine-tuned on an internal dataset from their institution. To handle potential data imbalance (more normal CXRs than pneumonia cases), they utilized techniques like data augmentation and weighted loss functions. The model was deployed in a virtual machine environment within the hospital's infrastructure, integrated with their existing Radiology Information System (RIS) and Picture Archiving and Communication System (PACS). The model processed CXRs and flagged those with suspected pneumonia, prioritizing them in the radiologists' worklist for expedited reporting. Diagnostic performance was evaluated using standard metrics like AUC, F1 score, sensitivity, and specificity, while the impact on clinical workflow was measured by report turnaround time (TAT).

Datasets

The study utilized three datasets:

  • Training Set: 4277 frontal CXRs from the National Centre for Infectious Diseases (NCID) and a partner hospital, with 971 ground truth-labeled as positive for pneumonia.
  • Proof-of-Concept (POC) Offsite Test Set: 1440 frontal CXRs from NCID, with 72 labeled as positive for pneumonia.
  • Clinical Deployment Set (Live Dataset): 3714 frontal CXRs acquired in NCID, with 189 labeled as positive for pneumonia.

The ground truth for pneumonia was established by two senior radiologists, with disagreements resolved through discussion and consultation with other radiologists.

Key Insights and Ideas

  • Real-world validation is crucial: The study emphasizes that AI models performing well in simulated environments may not translate directly to clinical practice. Rigorous testing and validation in a live setting are essential.
  • Ensemble models offer robustness: Combining multiple models (an ensemble) improved overall diagnostic performance compared to individual models, suggesting a more reliable and generalized prediction.
  • Prioritization accelerates workflow: By flagging abnormal CXRs, the AI model effectively prioritized cases for radiologists, leading to a significant reduction in report turnaround time. This is critical in high-volume screening scenarios.
  • Specificity is key for triage: The authors intentionally set a threshold to achieve high specificity, aiming to minimize false positives that could overwhelm radiologists and lead to unnecessary follow-ups, while still maintaining reasonable sensitivity.

Key Results

  • Diagnostic Performance (Clinical Deployment Set): The deployed model achieved an AUC of 0.9456, with a high specificity of 97.1% and a sensitivity of 78.8%. The F1 score was 0.9118.
  • Report Turnaround Time (TAT): The integration of the AI model reduced the average TAT by 22%, from nine minutes to seven minutes per report.
  • No Performance Degradation: Importantly, the study found no statistically significant degradation in diagnostic performance after clinical deployment, indicating the model's robustness.

Limitations and Future Work

  • Single-institution study: The model was developed, validated, and deployed within a single institution. Cross-institution deployment is suggested to ensure broader replicability and reliability.
  • Differentiating pneumonia types: The model's ability to reliably differentiate COVID-19 pneumonia from other community-acquired pneumonias based solely on CXRs remains a challenge, as imaging features can overlap.
  • Exclusion of clinical data: The current model relies solely on image data. Future work could incorporate non-image-related data (e.g., patient symptoms, lab results) to create a more comprehensive predictive model.
  • Paediatric population: The model was trained and tested on adult CXRs and cannot be extrapolated to the pediatric population.
  • Radiologist performance impact: The exact impact of AI augmentation on radiologist performance is still under investigation, with ongoing work in this area.

Summary updated

Abstract

(1) Background: Chest radiographs are the mainstay of initial radiological investigation in this COVID-19 pandemic. A reliable and readily deployable artificial intelligence (AI) algorithm that detects pneumonia in COVID-19 suspects can be useful for screening or triage in a hospital setting. This study has a few objectives: first, to develop a model that accurately detects pneumonia in COVID-19 suspects; second, to assess its performance in a real-world clinical setting; and third, by integrating the model with the daily clinical workflow, to measure its impact on report turn-around time. (2) Methods: The model was developed from the NIH Chest-14 open-source dataset and fine-tuned using an internal dataset comprising more than 4000 CXRs acquired in our institution. Input from two senior radiologists provided the reference standard. The model was integrated into daily clinical workflow, prioritising abnormal CXRs for expedited reporting. Area under the receiver operating characteristic curve (AUC), F1 score, sensitivity, and specificity were calculated to characterise diagnostic performance. The average time taken by radiologists in reporting the CXRs was compared against the mean baseline time taken prior to implementation of the AI model. (3) Results: 9431 unique CXRs were included in the datasets, of which 1232 were ground truth-labelled positive for pneumonia. On the "live" dataset, the model achieved an AUC of 0.95 (95% confidence interval (CI): 0.92, 0.96) corresponding to a specificity of 97% (95% CI: 0.97, 0.98) and sensitivity of 79% (95% CI: 0.72, 0.84). No statistically significant degradation of diagnostic performance was encountered during clinical deployment, and report turn-around time was reduced by 22%. (4) Conclusion: In real-world clinical deployment, our model expedites reporting of pneumonia in COVID-19 suspects while preserving diagnostic performance without significant model drift.
BibTeX
@inproceedings{sim2022diagnostic,
  author = {Jordan Sim and Yong Han Ting and Yuan Tang and Yangqin Feng and Xiaofeng Lei and Xiaohong Wang and Wenxiang Chen and Su Huang and Sum-Thai Wong and Zhongkang Lu and Yingnan Cui and Soo-Kng Teo and Xinxing Xu and Weimin Huang and Cher-Heng Tan},
  title = {{Diagnostic Performance of a Deep Learning Model Deployed at a National COVID-19 Screening Facility for Detection of Pneumonia on Frontal Chest Radiographs}},
  booktitle = {Healthcare},
  year = {2022},
  doi = {10.3390/healthcare10010175},
}
Figure